[jira] [Commented] (HIVE-27858) OOM happens when selecting many columns and JOIN.
[ https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794975#comment-17794975 ] John Sherman commented on HIVE-27858: - [~dkuzmenko] pointed out to me the HIVE-27856 has disabled CTE materialization to work around HIVE-24167 for the release. So it seems that the better effort here would to focus on the CBO plan explosion. > OOM happens when selecting many columns and JOIN. > -- > > Key: HIVE-27858 > URL: https://issues.apache.org/jira/browse/HIVE-27858 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0-beta-1 >Reporter: Ryu Kobayashi >Assignee: John Sherman >Priority: Critical > Labels: hive-4.0.0-must > Attachments: HIVE-27858.full.q, HIVE-27858.less.columns.q, ddl.sql, > query.sql > > > OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These > did not happen in Hive 2 previously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27858) OOM happens when selecting many columns and JOIN.
[ https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-27858: Attachment: HIVE-27858.less.columns.q HIVE-27858.full.q > OOM happens when selecting many columns and JOIN. > -- > > Key: HIVE-27858 > URL: https://issues.apache.org/jira/browse/HIVE-27858 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0-beta-1 >Reporter: Ryu Kobayashi >Assignee: John Sherman >Priority: Critical > Labels: hive-4.0.0-must > Attachments: HIVE-27858.full.q, HIVE-27858.less.columns.q, ddl.sql, > query.sql > > > OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These > did not happen in Hive 2 previously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27858) OOM happens when selecting many columns and JOIN.
[
https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794895#comment-17794895
]
John Sherman commented on HIVE-27858:
-
I did some preliminary investigation here -
if I remove all the columns except the ones being joined/operated on the query
it gets past the point of generating the initial CBO plan -
this plan is 307402 lines long. (I had to CTRL-C because it was taking over 10+
minutes)
{code:java}
apache/hive master > cat hive.plan.log | wc -l
307402 {code}
I thought maybe the plan is so complex due to the recursive withs and that the
CTEs are not being materialized, so I set:
{code:java}
set hive.optimize.cte.materialize.full.aggregate.only=false;
{code}
in hopes that it would generate a better plan.
It then fails much quicker with:
{code:java}
java.lang.RuntimeException: equivalence mapping violation
{code}
which is https://issues.apache.org/jira/browse/HIVE-24167
If I mechanically fix HIVE-24167 (I don't have the domain knowledge, to know if
the change is correct in all case) by:
{code:java}
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
index d7744587e6..5338bb61e0 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
@@ -3273,7 +3273,7 @@ private static Statistics applyRuntimeStats(Context
context, Statistics stats, O
PlanMapper pm = context.getPlanMapper();
OpTreeSignature treeSig = pm.getSignatureOf(op);
-pm.link(op, treeSig);
+pm.merge(op, treeSig);
StatsSource statsSource = context.getStatsSource();
if (!statsSource.canProvideStatsFor(op.getClass())) {
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
index 0823b6d9ba..0de4e83624 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
@@ -3767,8 +3767,7 @@ private Operator genFilterPlan(QB qb, ASTNode condn,
Operator input, boolean use
Operator output = putOpInsertMap(OperatorFactory.getAndMakeChild(
new FilterDesc(filterCond, false), new RowSchema(
inputRR.getColumnInfos()), input), inputRR);
-
-ctx.getPlanMapper().link(condn, output);
+ctx.getPlanMapper().merge(condn, output);
LOG.debug("Created Filter Plan for {} row schema: {}", qb.getId(),
inputRR.toString());
return output;
{code}
It does successfully compile and it also successfully compiles the query with
all the columns.
I likely do not have the cycles to further the investigation and build up the
domain knowledge to fix HIVE-24167 (or to investigate why the plan explodes
without "hive.optimize.cte.materialize.full.aggregate.only").
I've attached the test files for both less column version and full column
version.
However I will highlight it to some folks to see if they can spend some time
soon to investigate this JIRA and HIVE-24167.
> OOM happens when selecting many columns and JOIN.
> --
>
> Key: HIVE-27858
> URL: https://issues.apache.org/jira/browse/HIVE-27858
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0-beta-1
>Reporter: Ryu Kobayashi
>Assignee: John Sherman
>Priority: Critical
> Labels: hive-4.0.0-must
> Attachments: ddl.sql, query.sql
>
>
> OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These
> did not happen in Hive 2 previously.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-27858) OOM happens when selecting many columns and JOIN.
[ https://issues.apache.org/jira/browse/HIVE-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-27858 started by John Sherman. --- > OOM happens when selecting many columns and JOIN. > -- > > Key: HIVE-27858 > URL: https://issues.apache.org/jira/browse/HIVE-27858 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0-beta-1 >Reporter: Ryu Kobayashi >Assignee: John Sherman >Priority: Critical > Labels: hive-4.0.0-must > Attachments: ddl.sql, query.sql > > > OOM happens when executing [^query.sql] using a table in [^ddl.sql]. These > did not happen in Hive 2 previously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27563) Add typeof UDF
John Sherman created HIVE-27563: --- Summary: Add typeof UDF Key: HIVE-27563 URL: https://issues.apache.org/jira/browse/HIVE-27563 Project: Hive Issue Type: Improvement Components: UDF Reporter: John Sherman Assignee: John Sherman It would be useful to have a typeof UDF that returns a string describing the type of the argument. For example: SELECT typeof(int_column) FROM made_up_table would return: int This is useful for test purposes and debugging purposes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27358) Add test cases for HIVE-22816 (view result caching w/ transactional tables)
John Sherman created HIVE-27358: --- Summary: Add test cases for HIVE-22816 (view result caching w/ transactional tables) Key: HIVE-27358 URL: https://issues.apache.org/jira/browse/HIVE-27358 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: John Sherman Assignee: John Sherman HIVE-22816 fixed result caching of views containing transactional tables but did not add test cases. I've written some simple test cases to ensure this feature continues working. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-27298) Provide implementation of HMS thrift service that throws UnsupportedOperationException
[
https://issues.apache.org/jira/browse/HIVE-27298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17716473#comment-17716473
]
John Sherman edited comment on HIVE-27298 at 4/26/23 12:03 AM:
---
I'm inclined to go with:
a) since AbstractThriftHiveMetastore provides no value otherwise (users should
be implementing against ThriftHiveMetastore.Iface for similar guarantees that
AbstractThriftHiveMetastore today provides). The class name may be confusing,
but I also do not associate a class having Abstract in the name as being an
abstract class (others might though).
However, b has the advantage that it doesn't change the behavior that
downstream implementations may rely on (even if I think it is pointless).
Though, I work on one of the downstream implementations (IMPALA) and
AbstractThriftHiveMetastore was added to support our use case. I think Impala
is the only user of this class but I can't prove that.
This comment from AbstractThriftHiveMetastore illustrates the intent that the
current implementation actually does not provide:
{code}
/**
* This abstract class can be extended by any remote cache that implements HMS
APIs.
* The idea behind introducing this abstract class is not to break the build of
remote cache,
* whenever we add new HMS APIs.
*/
{code}
was (Author: jfs):
I'm inclined to go with:
a) since AbstractThriftHiveMetastore provides no value otherwise (users should
be implementing against ThriftHiveMetastore.Iface for similar guarantees that
AbstractThriftHiveMetastore today provides). The class name may be confusing,
but I also do not associate a class having Abstract in the name as being an
abstract class (others might though).
However, b has the advantage that it doesn't change the behavior that
downstream implementations may rely on (even if I think it is pointless).
Though, I work on one of the downstream implementations (IMPALA) and
AbstractThriftHiveMetastore was added to support our use case. I think Impala
is the only user of this class but I can't prove that.
> Provide implementation of HMS thrift service that throws
> UnsupportedOperationException
> --
>
> Key: HIVE-27298
> URL: https://issues.apache.org/jira/browse/HIVE-27298
> Project: Hive
> Issue Type: Improvement
>Reporter: John Sherman
>Priority: Major
>
> The intent of HIVE-25005 and AbstractThriftHiveMetastore class was to provide
> default implementation for every HMS thrift method that throws
> UnsupportedOperationException.
> However - HIVE-25005 made the class abstract, so as a result there are a
> variety of HMS service methods that are not implemented in
> AbstractThriftHiveMetastore.
> We should either:
> a) remove abstract from the class definition AbstractThriftHiveMetastore
> or
> b) add a new class that is not abstract.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27298) Provide implementation of HMS thrift service that throws UnsupportedOperationException
[ https://issues.apache.org/jira/browse/HIVE-27298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17716473#comment-17716473 ] John Sherman commented on HIVE-27298: - I'm inclined to go with: a) since AbstractThriftHiveMetastore provides no value otherwise (users should be implementing against ThriftHiveMetastore.Iface for similar guarantees that AbstractThriftHiveMetastore today provides). The class name may be confusing, but I also do not associate a class having Abstract in the name as being an abstract class (others might though). However, b has the advantage that it doesn't change the behavior that downstream implementations may rely on (even if I think it is pointless). Though, I work on one of the downstream implementations (IMPALA) and AbstractThriftHiveMetastore was added to support our use case. I think Impala is the only user of this class but I can't prove that. > Provide implementation of HMS thrift service that throws > UnsupportedOperationException > -- > > Key: HIVE-27298 > URL: https://issues.apache.org/jira/browse/HIVE-27298 > Project: Hive > Issue Type: Improvement >Reporter: John Sherman >Priority: Major > > The intent of HIVE-25005 and AbstractThriftHiveMetastore class was to provide > default implementation for every HMS thrift method that throws > UnsupportedOperationException. > However - HIVE-25005 made the class abstract, so as a result there are a > variety of HMS service methods that are not implemented in > AbstractThriftHiveMetastore. > We should either: > a) remove abstract from the class definition AbstractThriftHiveMetastore > or > b) add a new class that is not abstract. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27298) Provide implementation of HMS thrift service that throws UnsupportedOperationException
John Sherman created HIVE-27298: --- Summary: Provide implementation of HMS thrift service that throws UnsupportedOperationException Key: HIVE-27298 URL: https://issues.apache.org/jira/browse/HIVE-27298 Project: Hive Issue Type: Improvement Reporter: John Sherman The intent of HIVE-25005 and AbstractThriftHiveMetastore class was to provide default implementation for every HMS thrift method that throws UnsupportedOperationException. However - HIVE-25005 made the class abstract, so as a result there are a variety of HMS service methods that are not implemented in AbstractThriftHiveMetastore. We should either: a) remove abstract from the class definition AbstractThriftHiveMetastore or b) add a new class that is not abstract. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27285) Add TableMeta ownership for filterTableMetas
John Sherman created HIVE-27285:
---
Summary: Add TableMeta ownership for filterTableMetas
Key: HIVE-27285
URL: https://issues.apache.org/jira/browse/HIVE-27285
Project: Hive
Issue Type: Improvement
Components: Standalone Metastore
Affects Versions: 4.0.0
Reporter: John Sherman
Assignee: John Sherman
Currently TableMeta does not include ownership information which makes it
difficult for filterTableMetas to efficiently filter based on ${OWNER}
privileges.
We should add ownership information to TableMeta and utilizing it in
filterTableMetas authorization checks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27217) addWriteNotificationLogInBatch can silently fail
[ https://issues.apache.org/jira/browse/HIVE-27217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-27217: --- > addWriteNotificationLogInBatch can silently fail > > > Key: HIVE-27217 > URL: https://issues.apache.org/jira/browse/HIVE-27217 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > Debugging an issue, I noticed that addWriteNotificationLogInBatch in > Hive.java can fail silently if the TApplicationException thrown is not > TApplicationException.UNKNOWN_METHOD or > TApplicationException.WRONG_METHOD_NAME. > https://github.com/apache/hive/blob/40a7d689e51d02fa9b324553fd1810d0ad043080/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3359-L3381 > Failures to write in the notification log can be very difficult to debug, we > should rethrow the exception so that the failure is very visible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26938) Investigate SMB Map Join for FULL OUTER
[ https://issues.apache.org/jira/browse/HIVE-26938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-26938: --- > Investigate SMB Map Join for FULL OUTER > --- > > Key: HIVE-26938 > URL: https://issues.apache.org/jira/browse/HIVE-26938 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > HIVE-18908 added FULL OUTER Map Join support but this work did not add > support for SMB Map Joins for FULL OUTER. > We should investigate if we can safely support SMB Map Join for this scenario > and implement it if so. > This is the area in which it gives up conversion, if we modify this line to > pass a 2nd argument of true to getBigTableCandidates to enable > isFullOuterJoinSupported - it does successfully convert (but we need to > verify that execution does the correct thing). > [https://github.com/apache/hive/blob/03ad025ada776c0d359124c6342615f1983c1a94/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L482] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26875) Transaction conflict retry loop only executes once
[
https://issues.apache.org/jira/browse/HIVE-26875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman resolved HIVE-26875.
-
Resolution: Fixed
Committed to master
> Transaction conflict retry loop only executes once
> --
>
> Key: HIVE-26875
> URL: https://issues.apache.org/jira/browse/HIVE-26875
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Currently the "conflict retry loop" only executes once.
> [https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L264]
> The intent of this loop is to detect if a conflicting transaction has
> committed while we were waiting to acquire locks. If there is a conflicting
> transaction, it invalidates the snapshot, rolls-back the transaction, opens a
> new transaction and tries to re-acquire locks (and then recompile). It then
> checks again if a conflicting transaction has committed and if so, redoes the
> above steps again, up to HIVE_TXN_MAX_RETRYSNAPSHOT_COUNT times.
> However - isValidTxnState relies on getNonSharedLockedTable():
> [https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/ql/src/java/org/apache/hadoop/hive/ql/DriverTxnHandler.java#L422]
> which does:
> {code:java}
> private Set getNonSharedLockedTables() {
> if (CollectionUtils.isEmpty(driver.getContext().getHiveLocks())) {
> return Collections.emptySet(); // Nothing to check
> }{code}
> getHiveLocks gets populated by lockAndRespond... HOWEVER -
> compileInternal ends up calling compile which ends up calling
> preparForCompile which ends up calling prepareContext which ends up
> destroying the context with the information lockAndRespond populated. So when
> the loop executes after all of this, it will never detect a 2nd conflict
> because isValidTxnState will always return true (because it thinks there are
> no locked objects).
> This manifests as duplicate records being created during concurrent UPDATEs
> if a transaction get conflicted twice.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26875) Transaction conflict retry loop only executes once
[
https://issues.apache.org/jira/browse/HIVE-26875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-26875:
---
> Transaction conflict retry loop only executes once
> --
>
> Key: HIVE-26875
> URL: https://issues.apache.org/jira/browse/HIVE-26875
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Critical
>
> Currently the "conflict retry loop" only executes once.
> [https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L264]
> The intent of this loop is to detect if a conflicting transaction has
> committed while we were waiting to acquire locks. If there is a conflicting
> transaction, it invalidates the snapshot, rolls-back the transaction, opens a
> new transaction and tries to re-acquire locks (and then recompile). It then
> checks again if a conflicting transaction has committed and if so, redoes the
> above steps again, up to HIVE_TXN_MAX_RETRYSNAPSHOT_COUNT times.
> However - isValidTxnState relies on getNonSharedLockedTable():
> [https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/ql/src/java/org/apache/hadoop/hive/ql/DriverTxnHandler.java#L422]
> which does:
> {code:java}
> private Set getNonSharedLockedTables() {
> if (CollectionUtils.isEmpty(driver.getContext().getHiveLocks())) {
> return Collections.emptySet(); // Nothing to check
> }{code}
> getHiveLocks gets populated by lockAndRespond... HOWEVER -
> compileInternal ends up calling compile which ends up calling
> preparForCompile which ends up calling prepareContext which ends up
> destroying the context with the information lockAndRespond populated. So when
> the loop executes after all of this, it will never detect a 2nd conflict
> because isValidTxnState will always return true (because it thinks there are
> no locked objects).
> This manifests as duplicate records being created during concurrent UPDATEs
> if a transaction get conflicted twice.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26633) Make thrift max message size configurable
[
https://issues.apache.org/jira/browse/HIVE-26633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17621161#comment-17621161
]
John Sherman commented on HIVE-26633:
-
Thanks [~dengzh]!
Looping around to add some more detail to the JIRA (just in case someone runs
across this issue).
This patch added "hive.thrift.client.max.message.size" to make the limit
configurable for both SSL and SASL (there was concerns in my last comment I
wouldn't be able to fix SSL paths, but the patch was re-worked to fix those
paths too).
I picked a larger default of 1gb vs 100mb internal thrift default, so hopefully
very few installations would need to ever adjust the above setting.
> Make thrift max message size configurable
> -
>
> Key: HIVE-26633
> URL: https://issues.apache.org/jira/browse/HIVE-26633
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Since thrift >= 0.14, thrift now enforces max message sizes through a
> TConfiguration object as described here:
> [https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
> By default MaxMessageSize gets set to 100MB.
> As a result it is possible for HMS clients not to be able to retrieve certain
> metadata for tables with a large amount of partitions or other metadata.
> For example on a cluster configured with kerberos between hs2 and hms,
> querying a large table (10k partitions, 200 columns with names of 200
> characters) results in this backtrace:
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388)
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>
> at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:420)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:399)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult.read(PartitionsByExprResult.java:335)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
> {code}
> Making this configurable (and defaulting to a higher value) would allow these
> tables to still be accessible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26633) Make thrift max message size configurable
[
https://issues.apache.org/jira/browse/HIVE-26633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17618333#comment-17618333
]
John Sherman commented on HIVE-26633:
-
I've added a WIP patch that makes it configurable via HiveConf and JdbcConf.
https://github.com/apache/hive/pull/3674
One issue is that there seems to be no way to provide a different
TConfiguration object for SSL connections.
Hive uses
https://github.com/apache/thrift/blob/9207b28b7df5f8e01960154971e8de07094c91a8/lib/java/src/main/java/org/apache/thrift/transport/TSSLTransportFactory.java#L290
which does a "new TSocket(socket)".
https://github.com/apache/thrift/blob/9207b28b7df5f8e01960154971e8de07094c91a8/lib/java/src/main/java/org/apache/thrift/transport/TSocket.java#L58
I have not had a chance to test with SSL enabled. I will try to get that done.
We may need follow up work for this.
Another interesting observation - this enforcement doesn't get hit without SASL
and/maybe SSL enabled. I would expect the limit by Thrift to be enforced
unilaterally.
> Make thrift max message size configurable
> -
>
> Key: HIVE-26633
> URL: https://issues.apache.org/jira/browse/HIVE-26633
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Since thrift >= 0.14, thrift now enforces max message sizes through a
> TConfiguration object as described here:
> [https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
> By default MaxMessageSize gets set to 100MB.
> As a result it is possible for HMS clients not to be able to retrieve certain
> metadata for tables with a large amount of partitions or other metadata.
> For example on a cluster configured with kerberos between hs2 and hms,
> querying a large table (10k partitions, 200 columns with names of 200
> characters) results in this backtrace:
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388)
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>
> at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:420)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:399)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult.read(PartitionsByExprResult.java:335)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
> {code}
> Making this configurable (and defaulting to a higher value) would allow these
> tables to still be accessible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26633) Make thrift max message size configurable
[
https://issues.apache.org/jira/browse/HIVE-26633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-26633:
---
> Make thrift max message size configurable
> -
>
> Key: HIVE-26633
> URL: https://issues.apache.org/jira/browse/HIVE-26633
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
>
> Since thrift >= 0.14, thrift now enforces max message sizes through a
> TConfiguration object as described here:
> [https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
> By default MaxMessageSize gets set to 100MB.
> As a result it is possible for HMS clients not to be able to retrieve certain
> metadata for tables with a large amount of partitions or other metadata.
> For example on a cluster configured with kerberos between hs2 and hms,
> querying a large table (10k partitions, 200 columns with names of 200
> characters) results in this backtrace:
> {code:java}
> org.apache.thrift.transport.TTransportException: MaxMessageSize reached
> at
> org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
>
> at
> org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
>
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
> at
> org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
>
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
> at
> org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
>
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454)
>
> at
> org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388)
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248)
>
> at
> org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270)
>
> at
> org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
>
> at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:420)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:399)
>
> at
> org.apache.hadoop.hive.metastore.api.PartitionsByExprResult.read(PartitionsByExprResult.java:335)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
> {code}
> Making this configurable (and defaulting to a higher value) would allow these
> tables to still be accessible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26584) compressed_skip_header_footer_aggr.q is flaky
[
https://issues.apache.org/jira/browse/HIVE-26584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612693#comment-17612693
]
John Sherman commented on HIVE-26584:
-
Thanks [~ayushtkn] and [~zabetak] for helping to make the patch better.
I don't see any indication that the tests require external tables. But I agree
that I could be overlooking a detail and I don't functionally need to change
the test so significantly (even if I think the newer version is cleaner). I
went with just rmr the created directories to clean up the directories at the
end.
I did not add DROP IF EXISTS or rmr(s) before the creation since I find that
practice typically hides problems and sometimes causes hidden dependencies
between tests.
As for clearTablesCreatedDuringTests - it doesn't clean these files up because
it only cleans up tables under the configured warehouse directory. This test
case manually creates a location not under the warehouse directory so it
doesn't end up cleaning up the files. I could modify the
clearTablesCreatedDuringTests to clean up all directories mentioned in CREATE
EXTERNAL TABLE location clauses, but that could be risky since it could lead to
user files being accidentally removed with a misconfigured location clause and
I am not sure I would be able to add all the possible checks to prevent that.
In the future I think tests like this should load the data via LOAD DATA if
possible and not use a custom LOCATION clause so it gets cleaned up normally.
> compressed_skip_header_footer_aggr.q is flaky
> -
>
> Key: HIVE-26584
> URL: https://issues.apache.org/jira/browse/HIVE-26584
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> One of my PRs compressed_skip_header_footer_aggr.q was failing with
> unexpected diff. Such as:
> {code:java}
> TestMiniLlapLocalCliDriver.testCliDriver:62 Client Execution succeeded but
> contained differences (error code = 1) after executing
> compressed_skip_header_footer_aggr.q
> 69,71c69,70
> < 1 2019-12-31
> < 2 2018-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 89d87
> < NULL NULL
> 91c89
> < 2 2018-12-31
> ---
> > 2 2019-12-31
> 100c98
> < 1
> ---
> > 2
> 109c107
> < 1 2019-12-31
> ---
> > 2 2019-12-31
> 127,128c125,126
> < 1 2019-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 146a145
> > 2 2019-12-31
> 155c154
> < 1
> ---
> > 2 {code}
> Investigating it, it did not seem to fail when executed locally. Since I
> suspected test interference I searched for the tablenames/directories used
> and discovered empty_skip_header_footer_aggr.q which uses the same table
> names AND external directories.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26584) compressed_skip_header_footer_aggr.q is flaky
[
https://issues.apache.org/jira/browse/HIVE-26584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612422#comment-17612422
]
John Sherman commented on HIVE-26584:
-
After reading the tests in deeper detail, I've modified the posted patch and
modified both compressed_skip_header_footer_aggr.q and
empty_skip_header_footer_aggr.q to:
1) Changed all the EXTERNAL TABLEs to normal managed tables
2) Added DROPs to the test for the created tables (so the underlying data gets
removed normally)
3) Removed the dfs commands that created directories and copied the test data
4) Added LOAD DATA commands to populate the tables with the test data
5) I also gave the tables unique names between the tests (I find it easier to
have unique names for debugging).
I find this approach less error prone and less confusing (LOAD DATA is more
idiomatic). I saw nothing inherent to the test cases that required external
tables.
> compressed_skip_header_footer_aggr.q is flaky
> -
>
> Key: HIVE-26584
> URL: https://issues.apache.org/jira/browse/HIVE-26584
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> One of my PRs compressed_skip_header_footer_aggr.q was failing with
> unexpected diff. Such as:
> {code:java}
> TestMiniLlapLocalCliDriver.testCliDriver:62 Client Execution succeeded but
> contained differences (error code = 1) after executing
> compressed_skip_header_footer_aggr.q
> 69,71c69,70
> < 1 2019-12-31
> < 2 2018-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 89d87
> < NULL NULL
> 91c89
> < 2 2018-12-31
> ---
> > 2 2019-12-31
> 100c98
> < 1
> ---
> > 2
> 109c107
> < 1 2019-12-31
> ---
> > 2 2019-12-31
> 127,128c125,126
> < 1 2019-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 146a145
> > 2 2019-12-31
> 155c154
> < 1
> ---
> > 2 {code}
> Investigating it, it did not seem to fail when executed locally. Since I
> suspected test interference I searched for the tablenames/directories used
> and discovered empty_skip_header_footer_aggr.q which uses the same table
> names AND external directories.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-26584) compressed_skip_header_footer_aggr.q is flaky
[
https://issues.apache.org/jira/browse/HIVE-26584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612407#comment-17612407
]
John Sherman edited comment on HIVE-26584 at 10/3/22 6:19 PM:
--
After digging in deeper - You are correct, it is not a concurrent issue. It
just happened to be the easiest way to repro and I mistakenly thought it was
the root of the issue (before we had the containerized ptest framework, test
conflicts were somewhat common iirc).
Here is what is what I think is happening:
1. During PR testing TestMiniLlapLocalCliDriver tests get split into 32
different splits
[https://github.com/apache/hive/blob/master/itests/bin/generate-cli-splits.sh]
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestMiniLlapLocalCliDriver.java#L39]
(It codegens 32 new TestMiniLlapLocalCliDriver objects each with split0 -
split32 in the package name)
2. Test assignment for each split is handled via runtime introspection of the
class name:
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestMiniLlapLocalCliDriver.java#L43]
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/SplitSupport.java#L46]
in my PRs case:
empty_skip_header_footer_aggr.q gets assigned to split-7:
{code:java}
{code}
compressed_skip_header_footer_aggr.q gets assigned to split-4:
{code:java}
{code}
3. All test splits are split across 20 executors (not sure where this lives,
maybe Jenkins scripts)
split-7 and split-4 get assigned to the same "execution split" of 14
{code:java}
split-14/itests/qtest/target/surefire-reports/TEST-org.apache.hadoop.hive.cli.split7.TestMiniLlapLocalCliDriver.xml
144:
split-14/itests/qtest/target/surefire-reports/TEST-org.apache.hadoop.hive.cli.split4.TestMiniLlapLocalCliDriver.xml
165:
{code}
4. empty_skip_header_footer_aggr gets executed before
compressed_skip_header_footer_aggr (this can be seen above in that 144 is
before 165 in the test xml)
5. Both empty_skip_header_footer_aggr and compressed_skip_header_footer_aggr
create external tables with the data copied to the same location(s).
For example these locations get used in both tests:
${system:test.tmp.dir}/testcase1
${system:test.tmp.dir}/testcase2
since each test invocation ends up using the same path and the tmp directory is
not cleaned between tests this is where the conflict occurs.
6. empty_skip_header_footer_aggr includes rmr commands to cleanup the testcase1
and testcase2 directories.
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/ql/src/test/queries/clientpositive/empty_skip_header_footer_aggr.q#L6]
compressed_skip_header does not:
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/ql/src/test/queries/clientpositive/compressed_skip_header_footer_aggr.q#L1]
This also like explains why it is not reproducible via:
{code:java}
mvn test -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=compressed_skip_header_footer_aggr.q,empty_skip_header_footer_aggr.q
{code}
I think the order of the tests when executed this way is always
compressed_skip_header_footer_aggr.q and then empty_skip_header_footer_aggr.q
My fix ends up working because I give a unique location for each tests test
external data files.
I'll likely modify empty_skip_header_footer_aggr.q to remove the rmr's (because
the only thing it really does is to hide this problem) and give all the
files/directories unique names. We could like add a "unique external directory"
variable that is generated per testcase and cleaned up after each one (or some
other solution) but I think that is out of the scope of this ticket.
was (Author: jfs):
After digging in deeper - You are correct, it is not a concurrent issue. It
just happened to be the easiest way to repro and I mistakenly thought it was
the root of the issue (before we had the containerized ptest framework, test
conflicts were somewhat common iirc).
Here is what is what I think is happening:
1. During PR testing TestMiniLlapLocalCliDriver tests get split into 32
different splits
[https://github.com/apache/hive/blob/master/itests/bin/generate-cli-splits.sh]
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestMiniLlapLocalCliDriver.java#L39]
(It codegens 32 new TestMiniLlapLocalCliDriver objects each with split0 -
split32 in the package name)
2. Test assignment for each split is handled via runtime introspection of the
class name:
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestMiniLlapLocalCliDriver.java#L43]
[jira] [Commented] (HIVE-26584) compressed_skip_header_footer_aggr.q is flaky
[
https://issues.apache.org/jira/browse/HIVE-26584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612407#comment-17612407
]
John Sherman commented on HIVE-26584:
-
After digging in deeper - You are correct, it is not a concurrent issue. It
just happened to be the easiest way to repro and I mistakenly thought it was
the root of the issue (before we had the containerized ptest framework, test
conflicts were somewhat common iirc).
Here is what is what I think is happening:
1. During PR testing TestMiniLlapLocalCliDriver tests get split into 32
different splits
[https://github.com/apache/hive/blob/master/itests/bin/generate-cli-splits.sh]
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestMiniLlapLocalCliDriver.java#L39]
(It codegens 32 new TestMiniLlapLocalCliDriver objects each with split0 -
split32 in the package name)
2. Test assignment for each split is handled via runtime introspection of the
class name:
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestMiniLlapLocalCliDriver.java#L43]
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/SplitSupport.java#L46]
in my PRs case:
empty_skip_header_footer_aggr.q gets assigned to split-7:
{code:java}
{code}
compressed_skip_header_footer_aggr.q gets assigned to split-4:
{code:java}
{code}
3. All test splits are split across 20 executors (not sure where this lives,
maybe Jenkins scripts)
split-7 and split-4 get assigned to the same "execution split" of 14
{code:java}
split-14/itests/qtest/target/surefire-reports/TEST-org.apache.hadoop.hive.cli.split7.TestMiniLlapLocalCliDriver.xml
144:
split-14/itests/qtest/target/surefire-reports/TEST-org.apache.hadoop.hive.cli.split4.TestMiniLlapLocalCliDriver.xml
165:
{code}
4. empty_skip_header_footer_aggr gets executed before
compressed_skip_header_footer_aggr (this can be seen above in that 144 is
before 165 in the test xml)
5. Both empty_skip_header_footer_aggr and compressed_skip_header_footer_aggr
create external tables with the data copied to the same location(s).
For example these locations get used in both tests:
${system:test.tmp.dir}/testcase1
${system:test.tmp.dir}/testcase2
since each test invocation ends up using the same path and the tmp directory is
not cleaned between tests this is where the conflict occurs.
6. empty_skip_header_footer_aggr includes rmr commands to cleanup the testcase1
and testcase2 directories.
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/ql/src/test/queries/clientpositive/empty_skip_header_footer_aggr.q#L6]
compressed_skip_header does not:
[https://github.com/apache/hive/blob/4170e566143e6daa291654e97116199aa738377c/ql/src/test/queries/clientpositive/compressed_skip_header_footer_aggr.q#L1]
This also like explains why it is not reproducible via:
{code:java}
mvn test -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=compressed_skip_header_footer_aggr.q,empty_skip_header_footer_aggr.q
{code}
I think the order of the tests when executed this way is always
compressed_skip_header_footer_aggr.q and then empty_skip_header_footer_aggr.q
My fix ends up working because I give a unique location for each tests test
external data files.
I'll likely modify empty_skip_header_footer_aggr.q to remove the rmr's (because
the only thing the do is to hide this problem) and give all the
files/directories unique names. We could like add a "unique external directory"
variable that is generated per testcase and cleaned up after each one (or some
other solution) but I think that is out of the scope of this ticket.
> compressed_skip_header_footer_aggr.q is flaky
> -
>
> Key: HIVE-26584
> URL: https://issues.apache.org/jira/browse/HIVE-26584
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> One of my PRs compressed_skip_header_footer_aggr.q was failing with
> unexpected diff. Such as:
> {code:java}
> TestMiniLlapLocalCliDriver.testCliDriver:62 Client Execution succeeded but
> contained differences (error code = 1) after executing
> compressed_skip_header_footer_aggr.q
> 69,71c69,70
> < 1 2019-12-31
> < 2 2018-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 89d87
> < NULL NULL
> 91c89
> < 2 2018-12-31
> ---
> > 2 2019-12-31
> 100c98
> < 1
> ---
> > 2
> 109c107
> < 1 2019-12-31
> ---
> > 2 2019-12-31
> 127,128c125,126
> < 1 2019-12-31
> < 3 2017-12-31
> ---
> >
[jira] [Work started] (HIVE-26584) compressed_skip_header_footer_aggr.q is flaky
[
https://issues.apache.org/jira/browse/HIVE-26584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-26584 started by John Sherman.
---
> compressed_skip_header_footer_aggr.q is flaky
> -
>
> Key: HIVE-26584
> URL: https://issues.apache.org/jira/browse/HIVE-26584
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> One of my PRs compressed_skip_header_footer_aggr.q was failing with
> unexpected diff. Such as:
> {code:java}
> TestMiniLlapLocalCliDriver.testCliDriver:62 Client Execution succeeded but
> contained differences (error code = 1) after executing
> compressed_skip_header_footer_aggr.q
> 69,71c69,70
> < 1 2019-12-31
> < 2 2018-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 89d87
> < NULL NULL
> 91c89
> < 2 2018-12-31
> ---
> > 2 2019-12-31
> 100c98
> < 1
> ---
> > 2
> 109c107
> < 1 2019-12-31
> ---
> > 2 2019-12-31
> 127,128c125,126
> < 1 2019-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 146a145
> > 2 2019-12-31
> 155c154
> < 1
> ---
> > 2 {code}
> Investigating it, it did not seem to fail when executed locally. Since I
> suspected test interference I searched for the tablenames/directories used
> and discovered empty_skip_header_footer_aggr.q which uses the same table
> names AND external directories.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26584) compressed_skip_header_footer_aggr.q is flaky
[
https://issues.apache.org/jira/browse/HIVE-26584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611938#comment-17611938
]
John Sherman commented on HIVE-26584:
-
This is reproducible locally by executing empty_skip_header_footer_aggr.q and
compressed_skip_header_footer_aggr.q concurrently.
> compressed_skip_header_footer_aggr.q is flaky
> -
>
> Key: HIVE-26584
> URL: https://issues.apache.org/jira/browse/HIVE-26584
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
>
> One of my PRs compressed_skip_header_footer_aggr.q was failing with
> unexpected diff. Such as:
> {code:java}
> TestMiniLlapLocalCliDriver.testCliDriver:62 Client Execution succeeded but
> contained differences (error code = 1) after executing
> compressed_skip_header_footer_aggr.q
> 69,71c69,70
> < 1 2019-12-31
> < 2 2018-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 89d87
> < NULL NULL
> 91c89
> < 2 2018-12-31
> ---
> > 2 2019-12-31
> 100c98
> < 1
> ---
> > 2
> 109c107
> < 1 2019-12-31
> ---
> > 2 2019-12-31
> 127,128c125,126
> < 1 2019-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 146a145
> > 2 2019-12-31
> 155c154
> < 1
> ---
> > 2 {code}
> Investigating it, it did not seem to fail when executed locally. Since I
> suspected test interference I searched for the tablenames/directories used
> and discovered empty_skip_header_footer_aggr.q which uses the same table
> names AND external directories.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26584) compressed_skip_header_footer_aggr.q is flaky
[
https://issues.apache.org/jira/browse/HIVE-26584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-26584:
---
> compressed_skip_header_footer_aggr.q is flaky
> -
>
> Key: HIVE-26584
> URL: https://issues.apache.org/jira/browse/HIVE-26584
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
>
> One of my PRs compressed_skip_header_footer_aggr.q was failing with
> unexpected diff. Such as:
> {code:java}
> TestMiniLlapLocalCliDriver.testCliDriver:62 Client Execution succeeded but
> contained differences (error code = 1) after executing
> compressed_skip_header_footer_aggr.q
> 69,71c69,70
> < 1 2019-12-31
> < 2 2018-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 89d87
> < NULL NULL
> 91c89
> < 2 2018-12-31
> ---
> > 2 2019-12-31
> 100c98
> < 1
> ---
> > 2
> 109c107
> < 1 2019-12-31
> ---
> > 2 2019-12-31
> 127,128c125,126
> < 1 2019-12-31
> < 3 2017-12-31
> ---
> > 2 2019-12-31
> > 3 2019-12-31
> 146a145
> > 2 2019-12-31
> 155c154
> < 1
> ---
> > 2 {code}
> Investigating it, it did not seem to fail when executed locally. Since I
> suspected test interference I searched for the tablenames/directories used
> and discovered empty_skip_header_footer_aggr.q which uses the same table
> names AND external directories.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611012#comment-17611012
]
John Sherman commented on HIVE-26320:
-
I think it feels more correct to investigate the ETypeConverter change, since
it feels like it is at the root of the issue rather than fixing the issue at
the symptom level (but the change could have wide effect and may uncover
different issues). As far as the GenericUDFIn and why it works with non struct
vs varchar, it is due to:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIn.java#L159]
{code:java}
case PRIMITIVE: {
Object arg = ((PrimitiveObjectInspector) compareOI)
.getPrimitiveJavaObject(conversionHelper.convertIfNecessary(arguments[0].get(),
argumentOIs[0]));
if (compareOI.getTypeName().equals(serdeConstants.BINARY_TYPE_NAME)) {
arg = ByteBuffer.wrap((byte[]) arg);
}
if (constantInSet.contains(arg)) {
bw.set(true);
return bw;
}
break;
} {code}
The getPrimitiveJavaObject will end up convert Text to the appropriate type -
as shown here for HiveVarchar.
[https://github.com/apache/hive/blob/c19d56ec7429bfcfad92b62ac335dbf8177dab24/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableHiveVarcharObjectInspector.java#L48]
However, structs are handled here:
[https://github.com/apache/hive/blob/c19d56ec7429bfcfad92b62ac335dbf8177dab24/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIn.java#L186]
{code:java}
case STRUCT: {
Object value;
if (argumentOIs[0] instanceof ConstantObjectInspector) {
value = ((ConstantObjectInspector)
argumentOIs[0]).getWritableConstantValue();
} else {
value = conversionHelper.convertIfNecessary(arguments[0].get(),
argumentOIs[0]);
}
if (constantInSet.contains(((StructObjectInspector)
compareOI).getStructFieldsDataAsList(value))) {
bw.set(true);
return bw;
}
break;
} {code}
getStructFieldsDataAsList does not attempt to do any conversion and just
returns the data
[https://github.com/apache/hive/blob/c19d56ec7429bfcfad92b62ac335dbf8177dab24/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ArrayWritableObjectInspector.java#L193]
I also would have thought that the convertIfNecessary methods should have
converted the string/text values to the appropriate types but it does not
because the Parquet SerDe ObjectInspector is saying the row contains the
appropriately typed objects (even though everything is actually represented as
Text), so the conversion doesn't take place because the converters already
think the input and output object inspector schemas match.
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26574) hive.merge.tezfiles does not work with LLAP hive.llap.execution.mode = all
[ https://issues.apache.org/jira/browse/HIVE-26574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610687#comment-17610687 ] John Sherman commented on HIVE-26574: - The root cause seems to be that TaskGraphWalker does not walk the direct children of a ConditionalTask thus when the LLAP Decider gets executed on the TaskGraph it doesn't convert the conditional Merge Work. > hive.merge.tezfiles does not work with LLAP hive.llap.execution.mode = all > -- > > Key: HIVE-26574 > URL: https://issues.apache.org/jira/browse/HIVE-26574 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > set hive.execution.mode=llap; > set hive.llap.execution.mode=all; > set hive.merge.tezfiles=true; > with the above settings the Merge work generated does not get converted to > LLAP work and thus fails on platform that require LLAP. > This will either end up causing the query compilation to fail OR a > NullPointerException at execution time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26574) hive.merge.tezfiles does not work with LLAP hive.llap.execution.mode = all
[ https://issues.apache.org/jira/browse/HIVE-26574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-26574: --- > hive.merge.tezfiles does not work with LLAP hive.llap.execution.mode = all > -- > > Key: HIVE-26574 > URL: https://issues.apache.org/jira/browse/HIVE-26574 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0-alpha-2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > set hive.execution.mode=llap; > set hive.llap.execution.mode=all; > set hive.merge.tezfiles=true; > with the above settings the Merge work generated does not get converted to > LLAP work and thus fails on platform that require LLAP. > This will either end up causing the query compilation to fail OR a > NullPointerException at execution time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610300#comment-17610300
]
John Sherman commented on HIVE-26320:
-
So digging into this - the root cause seems to be that the Parquet SerDe will
return STRING/Text types for CHAR and VARCHAR.
The code in GenericUDFIn creates a Set<> containing the constant part of the IN
clause which also includes the correct types.
So the constant IN Set<> would have entries like:
struct which is basically a List
containing a HiveVarcharWritable and an IntWritable.
The Parquet reader seems to produce:
struct which is basically a List containing
a Text and a IntWritable.
So the constant IN set doesn't match the rows produced due to the types being
different.
I could likely fix this for the GenericUDFIn case - but I suspect there are
other areas in which this type difference causes different results. So I've
attempted to fix this at the SerDe deserialization level by converting the row
data from PARQUET to the appropriate types (if needed).
I am a bit concerned with if this has performance implication, though I am
hoping the conversion between the string types are relatively low overhead.
I've posted a WIP patch to see how it does on tests and I will likely
investigate a couple of other areas.
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26472) Concurrent UPDATEs can cause duplicate rows
[
https://issues.apache.org/jira/browse/HIVE-26472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman resolved HIVE-26472.
-
Resolution: Fixed
Reviewed and committed
> Concurrent UPDATEs can cause duplicate rows
> ---
>
> Key: HIVE-26472
> URL: https://issues.apache.org/jira/browse/HIVE-26472
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-1
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Critical
> Labels: pull-request-available
> Attachments: debug.diff
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Concurrent UPDATEs to the same table can cause duplicate rows when the
> following occurs:
> Two UPDATEs get assigned txnIds and writeIds like this:
> UPDATE #1 = txnId: 100 writeId: 50 <--- commits first
> UPDATE #2 = txnId: 101 writeId: 49
> To replicate the issue:
> I applied the attach debug.diff patch which adds hive.lock.sleep.writeid
> (which controls the amount to sleep before acquiring a writeId) and
> hive.lock.sleep.post.writeid (which controls the amount to sleep after
> acquiring a writeId).
> {code:java}
> CREATE TABLE test_update(i int) STORED AS ORC
> TBLPROPERTIES('transactional'="true");
> INSERT INTO test_update VALUES (1);
> Start two beeline connections.
> In connection #1 - run:
> set hive.driver.parallel.compilation = true;
> set hive.lock.sleep.writeid=5s;
> update test_update set i = 1 where i = 1;
> Wait one second and in connection #2 - run:
> set hive.driver.parallel.compilation = true;
> set hive.lock.sleep.post.writeid=10s;
> update test_update set i = 1 where i = 1;
> After both updates complete - it is likely that test_update contains two rows
> now.
> {code}
> HIVE-24211 seems to address the case when:
> UPDATE #1 = txnId: 100 writeId: 50
> UPDATE #2 = txnId: 101 writeId: 49 <--- commits first (I think this causes
> UPDATE #1 to detect the snapshot is out of date because commitedTxn > UPDATE
> #1s txnId)
> A possible work around is to set hive.driver.parallel.compilation = false,
> but this would only help in cases there is only one HS2 instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26472) Concurrent UPDATEs can cause duplicate rows
[
https://issues.apache.org/jira/browse/HIVE-26472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580529#comment-17580529
]
John Sherman commented on HIVE-26472:
-
The current purposed patch approach is to always re-allocate writeIds during
recompilation when the transaction is not out of date (so the transaction gets
re-used). If the transaction is out of date, this is already handled by a full
rollback and open of a transaction.
This involves clearing the writeId cache and adding an optional reallocate
boolean to allocateTableWriteIds (that defaults to false). The reason the
reallocate boolean is required, is that allocateTableWriteIds will return the
previously acquired writeIds for the transaction otherwise. So we need to
delete the old assigned writeIds and associate new ones with the txn.
> Concurrent UPDATEs can cause duplicate rows
> ---
>
> Key: HIVE-26472
> URL: https://issues.apache.org/jira/browse/HIVE-26472
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-1
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Critical
> Labels: pull-request-available
> Attachments: debug.diff
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Concurrent UPDATEs to the same table can cause duplicate rows when the
> following occurs:
> Two UPDATEs get assigned txnIds and writeIds like this:
> UPDATE #1 = txnId: 100 writeId: 50 <--- commits first
> UPDATE #2 = txnId: 101 writeId: 49
> To replicate the issue:
> I applied the attach debug.diff patch which adds hive.lock.sleep.writeid
> (which controls the amount to sleep before acquiring a writeId) and
> hive.lock.sleep.post.writeid (which controls the amount to sleep after
> acquiring a writeId).
> {code:java}
> CREATE TABLE test_update(i int) STORED AS ORC
> TBLPROPERTIES('transactional'="true");
> INSERT INTO test_update VALUES (1);
> Start two beeline connections.
> In connection #1 - run:
> set hive.driver.parallel.compilation = true;
> set hive.lock.sleep.writeid=5s;
> update test_update set i = 1 where i = 1;
> Wait one second and in connection #2 - run:
> set hive.driver.parallel.compilation = true;
> set hive.lock.sleep.post.writeid=10s;
> update test_update set i = 1 where i = 1;
> After both updates complete - it is likely that test_update contains two rows
> now.
> {code}
> HIVE-24211 seems to address the case when:
> UPDATE #1 = txnId: 100 writeId: 50
> UPDATE #2 = txnId: 101 writeId: 49 <--- commits first (I think this causes
> UPDATE #1 to detect the snapshot is out of date because commitedTxn > UPDATE
> #1s txnId)
> A possible work around is to set hive.driver.parallel.compilation = false,
> but this would only help in cases there is only one HS2 instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26472) Concurrent UPDATEs can cause duplicate rows
[
https://issues.apache.org/jira/browse/HIVE-26472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-26472:
---
Assignee: John Sherman
> Concurrent UPDATEs can cause duplicate rows
> ---
>
> Key: HIVE-26472
> URL: https://issues.apache.org/jira/browse/HIVE-26472
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-1
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Critical
> Labels: pull-request-available
> Attachments: debug.diff
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Concurrent UPDATEs to the same table can cause duplicate rows when the
> following occurs:
> Two UPDATEs get assigned txnIds and writeIds like this:
> UPDATE #1 = txnId: 100 writeId: 50 <--- commits first
> UPDATE #2 = txnId: 101 writeId: 49
> To replicate the issue:
> I applied the attach debug.diff patch which adds hive.lock.sleep.writeid
> (which controls the amount to sleep before acquiring a writeId) and
> hive.lock.sleep.post.writeid (which controls the amount to sleep after
> acquiring a writeId).
> {code:java}
> CREATE TABLE test_update(i int) STORED AS ORC
> TBLPROPERTIES('transactional'="true");
> INSERT INTO test_update VALUES (1);
> Start two beeline connections.
> In connection #1 - run:
> set hive.driver.parallel.compilation = true;
> set hive.lock.sleep.writeid=5s;
> update test_update set i = 1 where i = 1;
> Wait one second and in connection #2 - run:
> set hive.driver.parallel.compilation = true;
> set hive.lock.sleep.post.writeid=10s;
> update test_update set i = 1 where i = 1;
> After both updates complete - it is likely that test_update contains two rows
> now.
> {code}
> HIVE-24211 seems to address the case when:
> UPDATE #1 = txnId: 100 writeId: 50
> UPDATE #2 = txnId: 101 writeId: 49 <--- commits first (I think this causes
> UPDATE #1 to detect the snapshot is out of date because commitedTxn > UPDATE
> #1s txnId)
> A possible work around is to set hive.driver.parallel.compilation = false,
> but this would only help in cases there is only one HS2 instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26472) Concurrent UPDATEs can cause duplicate rows
[
https://issues.apache.org/jira/browse/HIVE-26472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman updated HIVE-26472:
Priority: Critical (was: Major)
> Concurrent UPDATEs can cause duplicate rows
> ---
>
> Key: HIVE-26472
> URL: https://issues.apache.org/jira/browse/HIVE-26472
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0-alpha-1
>Reporter: John Sherman
>Priority: Critical
> Attachments: debug.diff
>
>
> Concurrent UPDATEs to the same table can cause duplicate rows when the
> following occurs:
> Two UPDATEs get assigned txnIds and writeIds like this:
> UPDATE #1 = txnId: 100 writeId: 50 <--- commits first
> UPDATE #2 = txnId: 101 writeId: 49
> To replicate the issue:
> I applied the attach debug.diff patch which adds hive.lock.sleep.writeid
> (which controls the amount to sleep before acquiring a writeId) and
> hive.lock.sleep.post.writeid (which controls the amount to sleep after
> acquiring a writeId).
> {code:java}
> CREATE TABLE test_update(i int) STORED AS ORC
> TBLPROPERTIES('transactional'="true");
> INSERT INTO test_update VALUES (1);
> Start two beeline connections.
> In connection #1 - run:
> set hive.driver.parallel.compilation = true;
> set hive.lock.sleep.writeid=5s;
> update test_update set i = 1 where i = 1;
> Wait one second and in connection #2 - run:
> set hive.driver.parallel.compilation = true;
> set hive.lock.sleep.post.writeid=10s;
> update test_update set i = 1 where i = 1;
> After both updates complete - it is likely that test_update contains two rows
> now.
> {code}
> HIVE-24211 seems to address the case when:
> UPDATE #1 = txnId: 100 writeId: 50
> UPDATE #2 = txnId: 101 writeId: 49 <--- commits first (I think this causes
> UPDATE #1 to detect the snapshot is out of date because commitedTxn > UPDATE
> #1s txnId)
> A possible work around is to set hive.driver.parallel.compilation = false,
> but this would only help in cases there is only one HS2 instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26423) Make commons-pool2 an explicit dependency
[ https://issues.apache.org/jira/browse/HIVE-26423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570207#comment-17570207 ] John Sherman commented on HIVE-26423: - The patch makes the dependency version 2.11.1 -> the original version being transitively imported was 2.7.0. I went with the newest version since I saw no issues with it and to attempt to be safe from a CVE standpoint. > Make commons-pool2 an explicit dependency > - > > Key: HIVE-26423 > URL: https://issues.apache.org/jira/browse/HIVE-26423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > HIVE-26242 started using the commons-pool2 which is getting pulled in a > transitive dependency through commons-dbcp2 or calcite-core. It would be > better to make it an explicit dependency to ensure it gets packaged properly > and/or things do not suddenly break if the transitive dependencies change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26423) Make commons-pool2 an explicit dependency
[ https://issues.apache.org/jira/browse/HIVE-26423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-26423: Summary: Make commons-pool2 an explicit dependency (was: Make commons-pool2 a explicit dependency) > Make commons-pool2 an explicit dependency > - > > Key: HIVE-26423 > URL: https://issues.apache.org/jira/browse/HIVE-26423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > HIVE-26242 started using the commons-pool2 which is getting pulled in a > transitive dependency through commons-dbcp2 or calcite-core. It would be > better to make it an explicit dependency to ensure it gets packaged properly > and/or things do not suddenly break if the transitive dependencies change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26423) Make commons-pool2 a explicit dependency
[ https://issues.apache.org/jira/browse/HIVE-26423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-26423: --- > Make commons-pool2 a explicit dependency > > > Key: HIVE-26423 > URL: https://issues.apache.org/jira/browse/HIVE-26423 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > HIVE-26242 started using the commons-pool2 which is getting pulled in a > transitive dependency through commons-dbcp2 or calcite-core. It would be > better to make it an explicit dependency to ensure it gets packaged properly > and/or things do not suddenly break if the transitive dependencies change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26418) Stop packaging hive-standalone-metastore-server*-tests.jar
[
https://issues.apache.org/jira/browse/HIVE-26418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-26418:
---
> Stop packaging hive-standalone-metastore-server*-tests.jar
> --
>
> Key: HIVE-26418
> URL: https://issues.apache.org/jira/browse/HIVE-26418
> Project: Hive
> Issue Type: Bug
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
>
> HIVE-25989 added hive-standalone-metastore-server with the classifier of
> tests to hbase-handler/pom.xml but without the scope of test.As a result
> hive-standalone-metastore-server*-tests.jar gets packaged.
> {code:java}
> (packaging) $ find . -iname "*test*.jar"
> ./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-llap-common-4.0.0-alpha-2-SNAPSHOT-tests.jar
> ./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-testutils-4.0.0-alpha-2-SNAPSHOT.jar
> ./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-standalone-metastore-server-4.0.0-alpha-2-SNAPSHOT-tests.jar
> ./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/opentest4j-1.2.0.jar
> {code}
> Changing the pom file to use scope test for the tests jar and to also bring
> in the non-tests jar for runtime dependency fixes the issue:
> {code:java}
> (packaging) $ find . -iname "*test*.jar"
> ./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-llap-common-4.0.0-alpha-2-SNAPSHOT-tests.jar
> ./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-testutils-4.0.0-alpha-2-SNAPSHOT.jar
> ./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/opentest4j-1.2.0.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26395) Support CREATE TABLE LIKE FILE for PARQUET
[
https://issues.apache.org/jira/browse/HIVE-26395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman resolved HIVE-26395.
-
Fix Version/s: 4.0.0
Resolution: Fixed
Committed.
> Support CREATE TABLE LIKE FILE for PARQUET
> --
>
> Key: HIVE-26395
> URL: https://issues.apache.org/jira/browse/HIVE-26395
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The intent is to allow a user to create a table and derive the schema from a
> user provided parquet file. A secondary goal is to generalize this support so
> other SerDes/formats could implement the feature easily.
> The proposed syntax is:
> CREATE TABLE LIKE FILE 'path to file';
> Example being:
> {code:java}
> CREATE TABLE like_test_all_types LIKE FILE PARQUET
> '${system:test.tmp.dir}/test_all_types/00_0';{code}
> with partitioning
> {code}
> CREATE TABLE like_test_partitioning LIKE FILE PARQUET
> '${system:test.tmp.dir}/test_all_types/00_0' PARTITIONED BY (year STRING,
> month STRING);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568248#comment-17568248
]
John Sherman commented on HIVE-26320:
-
I'll play around with this some more. I had missed the detail about ORC/Text
working but not Parquet. My patch does fix it for parquet, but I would like to
know why ORC/Text file just worked? Maybe different handling of
varchar/char/string for the types in at the SerDe level?
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26320) Incorrect case evaluation for Parquet based table
[
https://issues.apache.org/jira/browse/HIVE-26320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-26320:
---
Assignee: John Sherman
> Incorrect case evaluation for Parquet based table
> -
>
> Key: HIVE-26320
> URL: https://issues.apache.org/jira/browse/HIVE-26320
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 4.0.0-alpha-1
>Reporter: Chiran Ravani
>Assignee: John Sherman
>Priority: Major
>
> Query involving case statement with two or more conditions leads to incorrect
> result for tables with parquet format, The problem is not observed with ORC
> or TextFile.
> *Steps to reproduce*:
> {code:java}
> create external table case_test_parquet(kob varchar(2),enhanced_type_code
> int) stored as parquet;
> insert into case_test_parquet values('BB',18),('BC',18),('AB',18);
> select case when (
>(kob='BB' and enhanced_type_code='18')
>or (kob='BC' and enhanced_type_code='18')
> )
> then 1
> else 0
> end as logic_check
> from case_test_parquet;
> {code}
> Result:
> {code}
> 0
> 0
> 0
> {code}
> Expected result:
> {code}
> 1
> 1
> 0
> {code}
> The problem does not appear when setting hive.optimize.point.lookup=false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26398) INSERT INTO complex type with varchar/char field fails CBO
[
https://issues.apache.org/jira/browse/HIVE-26398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-26398:
---
> INSERT INTO complex type with varchar/char field fails CBO
> --
>
> Key: HIVE-26398
> URL: https://issues.apache.org/jira/browse/HIVE-26398
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
>
> {code}
> CREATE EXTERNAL TABLE test_complex_struct(struct_type struct STRING, varchar_type : VARCHAR(100), char_type : CHAR(34)>) STORED AS PARQUET;
> INSERT INTO test_complex_struct SELECT named_struct("string_type", 'str1',
> "varchar_type", cast('varchar1' as varchar(100)), "char_type", cast('char' as
> char(34)));
> {code}
> fails with
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 2:12 Cannot insert
> into target table because column number/types are different
> 'test_complex_struct': Cannot convert column 0 from
> struct to
> struct.{code}
> This also fails with maps:
> {code}
> CREATE EXTERNAL TABLE test_complex_map(t_map map)
> STORED AS PARQUET;
> INSERT INTO test_complex_map SELECT map(cast("test" as varchar(100)),
> cast("value" as char(10)));
> {code}
> It seems that the cast is being lost or ignored somewhere along the way (and
> it stays a string).
> The queries do complete when ran on the non-CBO path. (Disabling CBO or
> falling back)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26395) Support CREATE TABLE LIKE FILE for PARQUET
[
https://issues.apache.org/jira/browse/HIVE-26395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17566895#comment-17566895
]
John Sherman commented on HIVE-26395:
-
Here is an older JIRA - https://issues.apache.org/jira/browse/HIVE-10593 with
discussion.
There is some conversation about using magic bytes to determine file format -
we did not go with this approach.
In the future it might be nice to have a feature that attempts to derive the
schema and filetype given a directory.
> Support CREATE TABLE LIKE FILE for PARQUET
> --
>
> Key: HIVE-26395
> URL: https://issues.apache.org/jira/browse/HIVE-26395
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The intent is to allow a user to create a table and derive the schema from a
> user provided parquet file. A secondary goal is to generalize this support so
> other SerDes/formats could implement the feature easily.
> The proposed syntax is:
> CREATE TABLE LIKE FILE 'path to file';
> Example being:
> {code:java}
> CREATE TABLE like_test_all_types LIKE FILE PARQUET
> '${system:test.tmp.dir}/test_all_types/00_0';{code}
> with partitioning
> {code}
> CREATE TABLE like_test_partitioning LIKE FILE PARQUET
> '${system:test.tmp.dir}/test_all_types/00_0' PARTITIONED BY (year STRING,
> month STRING);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26395) Support CREATE TABLE LIKE FILE for PARQUET
[
https://issues.apache.org/jira/browse/HIVE-26395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17566892#comment-17566892
]
John Sherman commented on HIVE-26395:
-
Impala has similar support and I used its implementation as a reference. Though
they do not have the FILE keyword in the syntax. I needed to add it to make the
syntax non-ambiguous since PARQUET/ORC/etc are not keywords in hive and I
wanted the option of third-party SerDes to be able to implement this.
> Support CREATE TABLE LIKE FILE for PARQUET
> --
>
> Key: HIVE-26395
> URL: https://issues.apache.org/jira/browse/HIVE-26395
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The intent is to allow a user to create a table and derive the schema from a
> user provided parquet file. A secondary goal is to generalize this support so
> other SerDes/formats could implement the feature easily.
> The proposed syntax is:
> CREATE TABLE LIKE FILE 'path to file';
> Example being:
> {code:java}
> CREATE TABLE like_test_all_types LIKE FILE PARQUET
> '${system:test.tmp.dir}/test_all_types/00_0';{code}
> with partitioning
> {code}
> CREATE TABLE like_test_partitioning LIKE FILE PARQUET
> '${system:test.tmp.dir}/test_all_types/00_0' PARTITIONED BY (year STRING,
> month STRING);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26395) Support CREATE TABLE LIKE FILE for PARQUET
[
https://issues.apache.org/jira/browse/HIVE-26395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-26395:
---
> Support CREATE TABLE LIKE FILE for PARQUET
> --
>
> Key: HIVE-26395
> URL: https://issues.apache.org/jira/browse/HIVE-26395
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
>
> The intent is to allow a user to create a table and derive the schema from a
> user provided parquet file. A secondary goal is to generalize this support so
> other SerDes/formats could implement the feature easily.
> The proposed syntax is:
> CREATE TABLE LIKE FILE 'path to file';
> Example being:
> {code:java}
> CREATE TABLE like_test_all_types LIKE FILE PARQUET
> '${system:test.tmp.dir}/test_all_types/00_0';{code}
> with partitioning
> {code}
> CREATE TABLE like_test_partitioning LIKE FILE PARQUET
> '${system:test.tmp.dir}/test_all_types/00_0' PARTITIONED BY (year STRING,
> month STRING);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-25768) Extend query-level HMS cache lifetime beyond analysis stage
[ https://issues.apache.org/jira/browse/HIVE-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-25768: Summary: Extend query-level HMS cache lifetime beyond analysis stage (was: Extended query-level HMS cache lifetime beyond analysis stage) > Extend query-level HMS cache lifetime beyond analysis stage > --- > > Key: HIVE-25768 > URL: https://issues.apache.org/jira/browse/HIVE-25768 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > HIVE-24176 added a HMS response cache to improve compilation times when > metadata is requested multiple times. The cache gets destroyed after > analysis. If we extend the lifetime of the cache beyond analysis it could be > used to reduce HMS communication for other areas such as various exec hooks > that inspect the metadata for a query. > The impetus that motivated this change is specifically the Atlas hook which > generates lineage information and makes HMS calls. These calls to HMS can add > latency to query results. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25768) Extended query-level HMS cache lifetime beyond analysis stage
[ https://issues.apache.org/jira/browse/HIVE-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-25768: --- > Extended query-level HMS cache lifetime beyond analysis stage > - > > Key: HIVE-25768 > URL: https://issues.apache.org/jira/browse/HIVE-25768 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > HIVE-24176 added a HMS response cache to improve compilation times when > metadata is requested multiple times. The cache gets destroyed after > analysis. If we extend the lifetime of the cache beyond analysis it could be > used to reduce HMS communication for other areas such as various exec hooks > that inspect the metadata for a query. > The impetus that motivated this change is specifically the Atlas hook which > generates lineage information and makes HMS calls. These calls to HMS can add > latency to query results. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25068) I need Hive access please
[ https://issues.apache.org/jira/browse/HIVE-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334817#comment-17334817 ] John Sherman commented on HIVE-25068: - Hi, Cassie. I suspect this isn't the proper forum for your request. It looks like maybe your company has an internal app called Hive in which you are having trouble accessing. This jira is for the Apache Hive project - [https://hive.apache.org/] which is an open source project. I think you likely need to reach out to a contact internal to your company. > I need Hive access please > - > > Key: HIVE-25068 > URL: https://issues.apache.org/jira/browse/HIVE-25068 > Project: Hive > Issue Type: Bug > Components: Authorization >Reporter: Cassie Moorhead >Priority: Minor > Attachments: Screen Shot 2021-04-28 at 10.41.36 AM.png > > > I am a TIP employee and just got an Upwork email and now can't access Hive > when I try to sign in using SSO. I keep getting the message: Sorry, you can't > access Hive because you are not assigned this app in Okta. If you're > wondering why this is happening, please [contact your > administrator|https://upworkcorp.okta.com/app/hive/exk1hrcixpqXq7XMv1d8/sso/saml?SAMLRequest=nVPBjtowEP2VyHcSEsIutYAVBbVFhTaCtFr1Uhl7UiwSO3gcNtuvrxNIy6HLgVOkmef3Zt68jJ%2FqIvdOYFBqNSGh3ydP0zGyIi%2FprLJ7tYFjBWg9B1NI28aEVEZRzVAiVawApJbT7Wy9opHfp6XRVnOdE2%2B5mJCfw4eHON6JSHBgUSSG2WCXEe97J%2BheOCBiBUuFlinrSv0o7PXjXjRKw5jGIY2G%2Frtw8IN4yYX6vVRCql%2B359idQUg%2FpWnSS75uU%2BIt3CZSMdtK760tkQZBVb5oc%2BDalL4%2BWOZzXQSsLIO9PEEA9SHcGy7r8vh8fHxen0IxChB10DhBvBkimIZurhVWBZgtmJPk8G2z%2BifgyPyGrGXu3gbx4xoy8Zri748wGK0%2BS3L2nbZumCvDb%2B%2FJuhHItBPsxMbBFWF31S%2BOYblIdC756z1X%2FaBNwezb6NAP24oUvayFUiiYzGdCGEB0luW5fpkbYBYmxJoKSNCNdskaiDZ5zlIL9V3Jm%2BuiZEZic2aoGbedt9fE89xZt4HsHqdvwjjlDbUrJ%2B7jwiWa4AJ3i6WGKSy1sZfT%2FG%2Be6bn3hh1%2Fu9d%2F5%2FQP=gUn50o3guHvFgPlA61EWMY6VoOiblF6z8nykJjPSkG3d0G6k6-QMsdCQhKbB3J0n4Ck8p5gMAdr9XeBCixKwSDPN-7dZjvA0F94g3yl9Mat5naZbhJBrbM99wm0j-hXJ9wTk6GJ-tKRRztC2RFo9w8uJwnQCouEnsoVOCIovxKsGNHZrxY5QUVew3WRqIJi7PU9otffzXVacvCHvQ1kLbupjZahgY2zhutaqZGJdMlytSoEQM2oY0zRAkuBjajZE14nfOcCT9VDWHq8iq6D6xPWxkHQ175NucEMmgyNIKkZlcVDb_-kpA9SJSOMELgc4KnYzF-Dqwu3DSGLBid6SMg#]. > If it's any consolation, we can take you to [your Okta home > page|https://upworkcorp.okta.com/]. > > I believe my email needs access. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-25068) I need Hive access please
[ https://issues.apache.org/jira/browse/HIVE-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman resolved HIVE-25068. - Resolution: Invalid > I need Hive access please > - > > Key: HIVE-25068 > URL: https://issues.apache.org/jira/browse/HIVE-25068 > Project: Hive > Issue Type: Bug > Components: Authorization >Reporter: Cassie Moorhead >Priority: Minor > Attachments: Screen Shot 2021-04-28 at 10.41.36 AM.png > > > I am a TIP employee and just got an Upwork email and now can't access Hive > when I try to sign in using SSO. I keep getting the message: Sorry, you can't > access Hive because you are not assigned this app in Okta. If you're > wondering why this is happening, please [contact your > administrator|https://upworkcorp.okta.com/app/hive/exk1hrcixpqXq7XMv1d8/sso/saml?SAMLRequest=nVPBjtowEP2VyHcSEsIutYAVBbVFhTaCtFr1Uhl7UiwSO3gcNtuvrxNIy6HLgVOkmef3Zt68jJ%2FqIvdOYFBqNSGh3ydP0zGyIi%2FprLJ7tYFjBWg9B1NI28aEVEZRzVAiVawApJbT7Wy9opHfp6XRVnOdE2%2B5mJCfw4eHON6JSHBgUSSG2WCXEe97J%2BheOCBiBUuFlinrSv0o7PXjXjRKw5jGIY2G%2Frtw8IN4yYX6vVRCql%2B359idQUg%2FpWnSS75uU%2BIt3CZSMdtK760tkQZBVb5oc%2BDalL4%2BWOZzXQSsLIO9PEEA9SHcGy7r8vh8fHxen0IxChB10DhBvBkimIZurhVWBZgtmJPk8G2z%2BifgyPyGrGXu3gbx4xoy8Zri748wGK0%2BS3L2nbZumCvDb%2B%2FJuhHItBPsxMbBFWF31S%2BOYblIdC756z1X%2FaBNwezb6NAP24oUvayFUiiYzGdCGEB0luW5fpkbYBYmxJoKSNCNdskaiDZ5zlIL9V3Jm%2BuiZEZic2aoGbedt9fE89xZt4HsHqdvwjjlDbUrJ%2B7jwiWa4AJ3i6WGKSy1sZfT%2FG%2Be6bn3hh1%2Fu9d%2F5%2FQP=gUn50o3guHvFgPlA61EWMY6VoOiblF6z8nykJjPSkG3d0G6k6-QMsdCQhKbB3J0n4Ck8p5gMAdr9XeBCixKwSDPN-7dZjvA0F94g3yl9Mat5naZbhJBrbM99wm0j-hXJ9wTk6GJ-tKRRztC2RFo9w8uJwnQCouEnsoVOCIovxKsGNHZrxY5QUVew3WRqIJi7PU9otffzXVacvCHvQ1kLbupjZahgY2zhutaqZGJdMlytSoEQM2oY0zRAkuBjajZE14nfOcCT9VDWHq8iq6D6xPWxkHQ175NucEMmgyNIKkZlcVDb_-kpA9SJSOMELgc4KnYzF-Dqwu3DSGLBid6SMg#]. > If it's any consolation, we can take you to [your Okta home > page|https://upworkcorp.okta.com/]. > > I believe my email needs access. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24645) UDF configure not called when fetch task conversion occurs
[ https://issues.apache.org/jira/browse/HIVE-24645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman resolved HIVE-24645. - Resolution: Fixed > UDF configure not called when fetch task conversion occurs > -- > > Key: HIVE-24645 > URL: https://issues.apache.org/jira/browse/HIVE-24645 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > When hive.fetch.task.conversion kicks in - UDF configure is not called. > This is likely due to MapredContext not being available when this conversion > occurs. > The approach I suggest is to create a dummy MapredContext and provide it with > the current configuration from ExprNodeGenericFuncEvaluator. > It is slightly unfortunate that the UDF API relies on MapredContext since > some aspects of the context do not apply to the variety of engines and > invocation paths for UDFs which makes it difficult to make a fully formed > dummy object such as the Reporter objects and the boolean around if it is a > Map context. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24645) UDF configure not called when fetch task conversion occurs
[ https://issues.apache.org/jira/browse/HIVE-24645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291725#comment-17291725 ] John Sherman commented on HIVE-24645: - This has been committed. > UDF configure not called when fetch task conversion occurs > -- > > Key: HIVE-24645 > URL: https://issues.apache.org/jira/browse/HIVE-24645 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > When hive.fetch.task.conversion kicks in - UDF configure is not called. > This is likely due to MapredContext not being available when this conversion > occurs. > The approach I suggest is to create a dummy MapredContext and provide it with > the current configuration from ExprNodeGenericFuncEvaluator. > It is slightly unfortunate that the UDF API relies on MapredContext since > some aspects of the context do not apply to the variety of engines and > invocation paths for UDFs which makes it difficult to make a fully formed > dummy object such as the Reporter objects and the boolean around if it is a > Map context. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24645) UDF configure not called when fetch task conversion occurs
[ https://issues.apache.org/jira/browse/HIVE-24645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-24645: --- > UDF configure not called when fetch task conversion occurs > -- > > Key: HIVE-24645 > URL: https://issues.apache.org/jira/browse/HIVE-24645 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > When hive.fetch.task.conversion kicks in - UDF configure is not called. > This is likely due to MapredContext not being available when this conversion > occurs. > The approach I suggest is to create a dummy MapredContext and provide it with > the current configuration from ExprNodeGenericFuncEvaluator. > It is slightly unfortunate that the UDF API relies on MapredContext since > some aspects of the context do not apply to the variety of engines and > invocation paths for UDFs which makes it difficult to make a fully formed > dummy object such as the Reporter objects and the boolean around if it is a > Map context. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-17684:
---
Assignee: Misha Dmitriev (was: John Sherman)
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Misha Dmitriev
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch, HIVE-17684.05.patch,
> HIVE-17684.06.patch, HIVE-17684.07.patch, HIVE-17684.08.patch,
> HIVE-17684.09.patch, HIVE-17684.10.patch, HIVE-17684.11.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-17684) HoS memory issues with MapJoinMemoryExhaustionHandler
[
https://issues.apache.org/jira/browse/HIVE-17684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-17684:
---
Assignee: John Sherman (was: Misha Dmitriev)
> HoS memory issues with MapJoinMemoryExhaustionHandler
> -
>
> Key: HIVE-17684
> URL: https://issues.apache.org/jira/browse/HIVE-17684
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: John Sherman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17684.01.patch, HIVE-17684.02.patch,
> HIVE-17684.03.patch, HIVE-17684.04.patch, HIVE-17684.05.patch,
> HIVE-17684.06.patch, HIVE-17684.07.patch, HIVE-17684.08.patch,
> HIVE-17684.09.patch, HIVE-17684.10.patch, HIVE-17684.11.patch
>
>
> We have seen a number of memory issues due the {{HashSinkOperator}} use of
> the {{MapJoinMemoryExhaustionHandler}}. This handler is meant to detect
> scenarios where the small table is taking too much space in memory, in which
> case a {{MapJoinMemoryExhaustionError}} is thrown.
> The configs to control this logic are:
> {{hive.mapjoin.localtask.max.memory.usage}} (default 0.90)
> {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} (default 0.55)
> The handler works by using the {{MemoryMXBean}} and uses the following logic
> to estimate how much memory the {{HashMap}} is consuming:
> {{MemoryMXBean#getHeapMemoryUsage().getUsed() /
> MemoryMXBean#getHeapMemoryUsage().getMax()}}
> The issue is that {{MemoryMXBean#getHeapMemoryUsage().getUsed()}} can be
> inaccurate. The value returned by this method returns all reachable and
> unreachable memory on the heap, so there may be a bunch of garbage data, and
> the JVM just hasn't taken the time to reclaim it all. This can lead to
> intermittent failures of this check even though a simple GC would have
> reclaimed enough space for the process to continue working.
> We should re-think the usage of {{MapJoinMemoryExhaustionHandler}} for HoS.
> In Hive-on-MR this probably made sense to use because every Hive task was run
> in a dedicated container, so a Hive Task could assume it created most of the
> data on the heap. However, in Hive-on-Spark there can be multiple Hive Tasks
> running in a single executor, each doing different things.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24592) Revert Hive-24550
[
https://issues.apache.org/jira/browse/HIVE-24592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259858#comment-17259858
]
John Sherman commented on HIVE-24592:
-
It might be nice if we could have a precommit check on the "passed tests" that
checked how stale the tag is and deny the commit if it was older than a few
days or a week (though some might find it annoying). I'm a little surprised
this situation doesn't happen more often.
> Revert Hive-24550
> -
>
> Key: HIVE-24592
> URL: https://issues.apache.org/jira/browse/HIVE-24592
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
>
> Broke the build.
>
> {code:none}
> [ERROR] COMPILATION ERROR :
> [ERROR]
> /home/travis/build/apache/hive/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2Acid.java:[23,44]
> cannot find symbol
> symbol: class TxnDbUtil
> location: package org.apache.hadoop.hive.metastore.txn
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:testCompile
> (default-testCompile) on project hive-it-unit: Compilation failure
> [ERROR]
> /home/travis/build/apache/hive/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHiveServer2Acid.java:[23,44]
> cannot find symbol
> [ERROR] symbol: class TxnDbUtil
> [ERROR] location: package org.apache.hadoop.hive.metastore.txn
> [ERROR]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24556) Optimize DefaultGraphWalker for case when node has no grandchildren
[ https://issues.apache.org/jira/browse/HIVE-24556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-24556: --- > Optimize DefaultGraphWalker for case when node has no grandchildren > --- > > Key: HIVE-24556 > URL: https://issues.apache.org/jira/browse/HIVE-24556 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > Investigating query with large IN clause with constant strings (100k+) taking > significant time during compilation revealed a possible optimization within > DefaultGraphWalker. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24549) TxnManager should not be shared across queries
[
https://issues.apache.org/jira/browse/HIVE-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-24549:
---
Assignee: (was: John Sherman)
> TxnManager should not be shared across queries
> --
>
> Key: HIVE-24549
> URL: https://issues.apache.org/jira/browse/HIVE-24549
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Priority: Major
>
> There are various sections of code that assume the DbTxnManager is not shared
> across concurrent queries in a session.
> Such as (which gets invoked during closeOperation):
>
> [https://github.com/apache/hive/blob/3f5e01cae5b65dde7edb3fbde8ebe70c1d02f6cf/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L868-L885]
> {code:java}
>// is usually called after close() to commit or rollback a query and end
> the driver life cycle.
> // do not understand why it is needed and wonder if it could be combined
> with close.
> @Override
> public void destroy() {
> driverState.lock();
> try {
> // in the cancel case where the driver state is INTERRUPTED, destroy
> will be deferred to
> // the query process
> if (driverState.isDestroyed()) {
> return;
> } else {
> driverState.descroyed();
> }
> } finally {
> driverState.unlock();
> }
> driverTxnHandler.destroy();
> }
> {code}
> The problematic part is the: driverTxnHandler.destroy() which looks like:
> {code:java}
> void destroy() {
>boolean isTxnOpen =
> driverContext != null &&
> driverContext.getTxnManager() != null &&
> driverContext.getTxnManager().isTxnOpen();
>release(!hiveLocks.isEmpty() || isTxnOpen);
> }
> {code}
> What happens is (rough sketch):
> Q1 - starts operation, acquires txn, does operation, closes txn/cleans up
> txn info, starts fetching data
> Q2 - starts operation, acquire txn
> Q1 - calls close operation which in turn calls destroy which sees the Q2s
> transaction information and cleans it up.
> Q2 - proceeds and fails in splitGeneration when it no longer can find its
> Valid*TxnIdList information.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24550) Cleanup only transaction information for the current DriverContext
[ https://issues.apache.org/jira/browse/HIVE-24550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-24550: --- > Cleanup only transaction information for the current DriverContext > -- > > Key: HIVE-24550 > URL: https://issues.apache.org/jira/browse/HIVE-24550 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > Long term solution would be: https://issues.apache.org/jira/browse/HIVE-24549 > Short term solution for the common usage pattern described in HIVE-24549 is > to ensure the current driverContext queryId matches the TxnManagers queryId. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-24549) TxnManager should not be shared across queries
[
https://issues.apache.org/jira/browse/HIVE-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250693#comment-17250693
]
John Sherman commented on HIVE-24549:
-
cc: [~pvargacl] did some investigation and was able to create a nice repro test
> TxnManager should not be shared across queries
> --
>
> Key: HIVE-24549
> URL: https://issues.apache.org/jira/browse/HIVE-24549
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
>
> There are various sections of code that assume the DbTxnManager is not shared
> across concurrent queries in a session.
> Such as (which gets invoked during closeOperation):
>
> [https://github.com/apache/hive/blob/3f5e01cae5b65dde7edb3fbde8ebe70c1d02f6cf/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L868-L885]
> {code:java}
>// is usually called after close() to commit or rollback a query and end
> the driver life cycle.
> // do not understand why it is needed and wonder if it could be combined
> with close.
> @Override
> public void destroy() {
> driverState.lock();
> try {
> // in the cancel case where the driver state is INTERRUPTED, destroy
> will be deferred to
> // the query process
> if (driverState.isDestroyed()) {
> return;
> } else {
> driverState.descroyed();
> }
> } finally {
> driverState.unlock();
> }
> driverTxnHandler.destroy();
> }
> {code}
> The problematic part is the: driverTxnHandler.destroy() which looks like:
> {code:java}
> void destroy() {
>boolean isTxnOpen =
> driverContext != null &&
> driverContext.getTxnManager() != null &&
> driverContext.getTxnManager().isTxnOpen();
>release(!hiveLocks.isEmpty() || isTxnOpen);
> }
> {code}
> What happens is (rough sketch):
> Q1 - starts operation, acquires txn, does operation, closes txn/cleans up
> txn info, starts fetching data
> Q2 - starts operation, acquire txn
> Q1 - calls close operation which in turn calls destroy which sees the Q2s
> transaction information and cleans it up.
> Q2 - proceeds and fails in splitGeneration when it no longer can find its
> Valid*TxnIdList information.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-24549) TxnManager should not be shared across queries
[
https://issues.apache.org/jira/browse/HIVE-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Sherman reassigned HIVE-24549:
---
> TxnManager should not be shared across queries
> --
>
> Key: HIVE-24549
> URL: https://issues.apache.org/jira/browse/HIVE-24549
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
>
> There are various sections of code that assume the DbTxnManager is not shared
> across concurrent queries in a session.
> Such as (which gets invoked during closeOperation):
>
> [https://github.com/apache/hive/blob/3f5e01cae5b65dde7edb3fbde8ebe70c1d02f6cf/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L868-L885]
> {code:java}
>// is usually called after close() to commit or rollback a query and end
> the driver life cycle.
> // do not understand why it is needed and wonder if it could be combined
> with close.
> @Override
> public void destroy() {
> driverState.lock();
> try {
> // in the cancel case where the driver state is INTERRUPTED, destroy
> will be deferred to
> // the query process
> if (driverState.isDestroyed()) {
> return;
> } else {
> driverState.descroyed();
> }
> } finally {
> driverState.unlock();
> }
> driverTxnHandler.destroy();
> }
> {code}
> The problematic part is the: driverTxnHandler.destroy() which looks like:
> {code:java}
> void destroy() {
>boolean isTxnOpen =
> driverContext != null &&
> driverContext.getTxnManager() != null &&
> driverContext.getTxnManager().isTxnOpen();
>release(!hiveLocks.isEmpty() || isTxnOpen);
> }
> {code}
> What happens is (rough sketch):
> Q1 - starts operation, acquires txn, does operation, closes txn/cleans up
> txn info, starts fetching data
> Q2 - starts operation, acquire txn
> Q1 - calls close operation which in turn calls destroy which sees the Q2s
> transaction information and cleans it up.
> Q2 - proceeds and fails in splitGeneration when it no longer can find its
> Valid*TxnIdList information.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-24086) CTAS with HMS translation enabled returns empty results.
[ https://issues.apache.org/jira/browse/HIVE-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186076#comment-17186076 ] John Sherman commented on HIVE-24086: - I suspect HIVE-23968 is the same issue or very similar. > CTAS with HMS translation enabled returns empty results. > > > Key: HIVE-24086 > URL: https://issues.apache.org/jira/browse/HIVE-24086 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > when you execute something like > create table ctas_table as select * from mgd_table; > if mgd_table is a managed table, the hive query planner creates a plan with > ctas_table as a managed table, so the location is set to something in the > managed warehouse directory. > However with HMS translation enabled, non-acid MANAGED tables are converted > to EXTERNAL with purge set to true. So the table location for this table is > altered to be in the external warehouse directory. > But after the table creation, the rest of the query executes but the data is > copied to the location set in the query plan. As a result when you execute a > select from ctas_table, it will not return any results because that location > is empty. > Workarounds: > 1) if hive.create.as.acid=true, or hive.create.as.insert.only=true, this > ctas_table remains a MANAGED acid table and HMS does not translate the table > to an EXTERNAL table. > 2) Instead of a create table as select, use >a) Create EXTERNAL table ctas_table as select * from mgd_table to make the > target table an external table > OR >b) Create transactional table ctas_table as select * from mgd_table to > make the target table an managed acid table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-19546) Enable TestStats#partitionedTableDeprecatedCalls, TestStats#partitionedTableInHiveCatalog, and TestStats#partitionedTableOtherCatalog
[ https://issues.apache.org/jira/browse/HIVE-19546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-19546: --- Assignee: John Sherman > Enable TestStats#partitionedTableDeprecatedCalls, > TestStats#partitionedTableInHiveCatalog, and > TestStats#partitionedTableOtherCatalog > - > > Key: HIVE-19546 > URL: https://issues.apache.org/jira/browse/HIVE-19546 > Project: Hive > Issue Type: Test > Components: Test >Affects Versions: 3.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: John Sherman >Priority: Critical > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23699) Cleanup HIVEQUERYRESULTFILEFORMAT handling
[ https://issues.apache.org/jira/browse/HIVE-23699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-23699: --- > Cleanup HIVEQUERYRESULTFILEFORMAT handling > -- > > Key: HIVE-23699 > URL: https://issues.apache.org/jira/browse/HIVE-23699 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > HIVEQUERYRESULTFILEFORMAT format handling has grown over the years and has > become somewhat messy code wise in SemanticAnalyzer and TaskCompiler. There > are special cases where the HIVEQUERYRESULTFILEFORMAT setting gets changed at > runtime that may cause issues if a user changes execution.engines between > queries and probably other corner cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23614) Always pass HiveConfig to removeTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126283#comment-17126283 ] John Sherman commented on HIVE-23614: - [~jcamachorodriguez] This is my first time using the new PR based tests. I'm not sure what to expect. It did do this: [https://travis-ci.org/github/apache/hive/builds/694812403] but I don't see any test results and it seemed kind of quick. > Always pass HiveConfig to removeTempOrDuplicateFiles > > > Key: HIVE-23614 > URL: https://issues.apache.org/jira/browse/HIVE-23614 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > As part of HIVE-23354, we check the provided HiveConf for speculative > execution and throw an error if it is enabled. There is one path that did not > previously provide a HiveConf value which shows up in test failures in > runtime_skewjoin_mapjoin_spark.q, skewjoin.q and skewjoin_onesideskew.q > (which we do not run as part of pre-commit tests) or at least not through > TestCliDriver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23614) Always pass HiveConfig to removeTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23614: Description: As part of HIVE-23354, we check the provided HiveConf for speculative execution and throw an error if it is enabled. There is one path that did not previously provide a HiveConf value which shows up in test failures in runtime_skewjoin_mapjoin_spark.q, skewjoin.q and skewjoin_onesideskew.q (which we do not run as part of pre-commit tests) or at least not through TestCliDriver. (was: As part of HIVE-23354, we check the provided HiveConf for speculative execution and throw an error if it is enabled. There is one path that did not previously provide a HiveConf value which shows up in test failures in runtime_skewjoin_mapjoin_spark.q, skewjoin.q and skewjoin_onesideskew.q (which we do not run as part of pre-commit tests).) > Always pass HiveConfig to removeTempOrDuplicateFiles > > > Key: HIVE-23614 > URL: https://issues.apache.org/jira/browse/HIVE-23614 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > As part of HIVE-23354, we check the provided HiveConf for speculative > execution and throw an error if it is enabled. There is one path that did not > previously provide a HiveConf value which shows up in test failures in > runtime_skewjoin_mapjoin_spark.q, skewjoin.q and skewjoin_onesideskew.q > (which we do not run as part of pre-commit tests) or at least not through > TestCliDriver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23614) Always pass HiveConfig to removeTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-23614: --- > Always pass HiveConfig to removeTempOrDuplicateFiles > > > Key: HIVE-23614 > URL: https://issues.apache.org/jira/browse/HIVE-23614 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > As part of HIVE-23354, we check the provided HiveConf for speculative > execution and throw an error if it is enabled. There is one path that did not > previously provide a HiveConf value which shows up in test failures in > runtime_skewjoin_mapjoin_spark.q, skewjoin.q and skewjoin_onesideskew.q > (which we do not run as part of pre-commit tests). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Status: Patch Available (was: Reopened) > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch, > HIVE-23478.3.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Attachment: HIVE-23478.3.patch > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch, > HIVE-23478.3.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109584#comment-17109584 ] John Sherman commented on HIVE-23478: - Incorporated changes from HIVE-23484 > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch, > HIVE-23478.3.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109555#comment-17109555 ] John Sherman commented on HIVE-23478: - Actually, I think I confused myself with similar named tests. I'll update this patch with the ordering change. > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reopened HIVE-23478: - > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman resolved HIVE-23478. - Resolution: Duplicate Closing as duplicate of HIVE-23484 > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Status: Open (was: Patch Available) > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109550#comment-17109550 ] John Sherman commented on HIVE-23478: - [~ashutoshc] I was just looking at that one, I do believe that is the correct approach. I'm going to suggest making the DB name on that patch unique also. I had seen problems running both test together locally which I surmised to be the DB name, but I didn't repro the ordering formatting change due to order. I'll abandon this one. > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Attachment: HIVE-23478.2.patch > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch, HIVE-23478.2.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Attachment: HIVE-23354.7.patch > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch, > HIVE-23354.6.patch, HIVE-23354.7.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109528#comment-17109528 ] John Sherman commented on HIVE-23354: - Re-uploaded patch in hopes test infra runs this time. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch, > HIVE-23354.6.patch, HIVE-23354.7.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108716#comment-17108716 ] John Sherman commented on HIVE-23354: - I've created https://issues.apache.org/jira/browse/HIVE-23478 for the spurious test failure. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch, HIVE-23354.6.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[
https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108715#comment-17108715
]
John Sherman commented on HIVE-23478:
-
I've attempted to fix the issue by making the database unique. I've changed the
database used from
{code:java}
db~!@#$%^&*(),<>{code}
to
{code:java}
db~!@@#$%^&*(),<>{code}
(I've added an extra @)
> Fix flaky special_character_in_tabnames_quotes_1 test
> -
>
> Key: HIVE-23478
> URL: https://issues.apache.org/jira/browse/HIVE-23478
> Project: Hive
> Issue Type: Improvement
> Components: Tests
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23478.1.patch
>
>
> While testing https://issues.apache.org/jira/browse/HIVE-23354
> special_character_in_tabnames_quotes_1 failed. Searching for the test, it
> seems other patches have also had failures. I noticed that
> special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1
> use the same database/table names. I suspect this is responsible for some of
> the flakiness.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Attachment: HIVE-23478.1.patch > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23478: Status: Patch Available (was: Open) > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23478.1.patch > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
[ https://issues.apache.org/jira/browse/HIVE-23478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-23478: --- > Fix flaky special_character_in_tabnames_quotes_1 test > - > > Key: HIVE-23478 > URL: https://issues.apache.org/jira/browse/HIVE-23478 > Project: Hive > Issue Type: Improvement > Components: Tests >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > > While testing https://issues.apache.org/jira/browse/HIVE-23354 > special_character_in_tabnames_quotes_1 failed. Searching for the test, it > seems other patches have also had failures. I noticed that > special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 > use the same database/table names. I suspect this is responsible for some of > the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Attachment: HIVE-23354.6.patch > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch, HIVE-23354.6.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Attachment: HIVE-23354.5.patch > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch, HIVE-23354.5.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106792#comment-17106792 ] John Sherman commented on HIVE-23354: - Newest patch I expect to do much better on the test front, I took a different approach on disabling speculative execution. (Similar to how hive.mapred.reduce.tasks.speculative.execution worked, but in reverse). I also marked hive.mapred.reduce.tasks.speculative.execution as deprecated and removed the usage of it, to prevent foot shooting. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Attachment: HIVE-23354.4.patch > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch, HIVE-23354.4.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106758#comment-17106758 ] John Sherman commented on HIVE-23354: - I expect some failures with the new commit. I'm still trying to work out the best way to handle the fact that mapreduce.map.speculative is the default setting from MR config. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Attachment: HIVE-23354.3.patch > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch, > HIVE-23354.3.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105863#comment-17105863 ] John Sherman commented on HIVE-23354: - I've adjusted the patch to check for speculative execution and throw an exception. After discussion with other Hive contributors/committers, MR is not as widely used with newer version of Hive and speculative execution with Tez seems to be marked as unstable. Especially since this code as it currently exists is also broken for speculative execution. A follow up Jira will be made once this patch gets reviewed to look into supporting speculative execution in this path. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Attachment: HIVE-23354.2.patch > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch, HIVE-23354.2.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102779#comment-17102779
]
John Sherman commented on HIVE-23182:
-
[~belugabehr] I think that example is the one my patch fixed.
If we make changes around this area we might also want to fix or inform Apache
Impala as we do so.
[https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java#L143]
They invoke the HiveLexer but do not set HiveConf. So if we make it a
Precondition, usages like this will fail.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23182) Semantic Exception: rule Identifier failed predicate allowQuotedId
[
https://issues.apache.org/jira/browse/HIVE-23182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102759#comment-17102759
]
John Sherman commented on HIVE-23182:
-
[~jcamachorodriguez] I think it would be solved by my JIRA. There may be other
areas where Parse gets called without HiveConf that I did not have a chance to
closely analyze.
It may be useful to have a new JIRA for [~belugabehr]'s idea of making it
default to be true in the lexer since this is an easy mistake to make.
> Semantic Exception: rule Identifier failed predicate allowQuotedId
> --
>
> Key: HIVE-23182
> URL: https://issues.apache.org/jira/browse/HIVE-23182
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Priority: Major
> Attachments: Querying a Hive Table (via Hiveserver2) with Colum... -
> Cloudera Community.pdf
>
>
> Querying a Hive Table (via Hiveserver2) with Column Masking enabled via
> Ranger Hive Plugin returns with an error.
> {code:none}
> [42000]: Error while compiling statement: FAILED: SemanticException
> org.apache.hadoop.hive.ql.parse.ParseException: line 1:62 rule Identifier
> failed predicate: {allowQuotedId()}? line 1:74 rule Identifier failed
> predicate: {allowQuotedId()}? line 1:94 rule Identifier failed predicate:
> {allowQuotedId()}? line 1:117 rule Identifier failed predicate:
> {allowQuotedId()}?
> {code}
> https://community.cloudera.com/t5/Support-Questions/Querying-a-Hive-Table-via-Hiveserver2-with-Column-Masking/td-p/167260
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101295#comment-17101295 ] John Sherman commented on HIVE-23354: - [~kuczoram] So it would be my understanding that the current code also doesn't work with speculative execution. Is that a correct reading of the situation? Since it relies on attemptIds but will throw an error if the file size of the newest attempt not >= than the older attempts. I just want to make sure I'm not missing a nuance of the existing code path. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098985#comment-17098985 ] John Sherman commented on HIVE-23354: - [~kuczoram] Thanks for the feedback. I'll see if I can come up with a better solution. This is exactly why I didn't rush this through the commit process since there are so many cases to consider. I definitely don't want to introduce any regressions here since it is such a relied on part. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23357) Allow queue user to be configured separately from doAs
[ https://issues.apache.org/jira/browse/HIVE-23357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23357: Status: Patch Available (was: In Progress) > Allow queue user to be configured separately from doAs > -- > > Key: HIVE-23357 > URL: https://issues.apache.org/jira/browse/HIVE-23357 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23357.1.patch > > > There are cases where we may want to submit to a YARN queue as the logged in > user but not require full doas for writing of HDFS or authorization. > A setting could be added to control this behavior: > set hive.server2.tez.queue.access.method=default > default is to follow doas setting (the current method) > loggedin is to always username of logged in user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23357) Allow queue user to be configured separately from doAs
[ https://issues.apache.org/jira/browse/HIVE-23357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23357: Attachment: HIVE-23357.1.patch > Allow queue user to be configured separately from doAs > -- > > Key: HIVE-23357 > URL: https://issues.apache.org/jira/browse/HIVE-23357 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23357.1.patch > > > There are cases where we may want to submit to a YARN queue as the logged in > user but not require full doas for writing of HDFS or authorization. > A setting could be added to control this behavior: > set hive.server2.tez.queue.access.method=default > default is to follow doas setting (the current method) > loggedin is to always username of logged in user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23357) Allow queue user to be configured separately from doAs
[ https://issues.apache.org/jira/browse/HIVE-23357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23357 started by John Sherman. --- > Allow queue user to be configured separately from doAs > -- > > Key: HIVE-23357 > URL: https://issues.apache.org/jira/browse/HIVE-23357 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23357.1.patch > > > There are cases where we may want to submit to a YARN queue as the logged in > user but not require full doas for writing of HDFS or authorization. > A setting could be added to control this behavior: > set hive.server2.tez.queue.access.method=default > default is to follow doas setting (the current method) > loggedin is to always username of logged in user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23357) Allow queue user to be configured separately from doAs
[ https://issues.apache.org/jira/browse/HIVE-23357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23357: Description: There are cases where we may want to submit to a YARN queue as the logged in user but not require full doas for writing of HDFS or authorization. A setting could be added to control this behavior: set hive.server2.tez.queue.access.method=default default is to follow doas setting (the current method) loggedin is to always username of logged in user. was: There are cases where we may want to submit to a YARN queue as the logged in user but not require full doas for writing of HDFS or authorization. A setting could be added to control this behavior: set hive.server2.tez.queue.access.username.method=default default is to follow doas setting (the current method) loggedin is to always username of logged in user. > Allow queue user to be configured separately from doAs > -- > > Key: HIVE-23357 > URL: https://issues.apache.org/jira/browse/HIVE-23357 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > There are cases where we may want to submit to a YARN queue as the logged in > user but not require full doas for writing of HDFS or authorization. > A setting could be added to control this behavior: > set hive.server2.tez.queue.access.method=default > default is to follow doas setting (the current method) > loggedin is to always username of logged in user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23357) Allow queue user to be configured separately from doAs
[ https://issues.apache.org/jira/browse/HIVE-23357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-23357: --- > Allow queue user to be configured separately from doAs > -- > > Key: HIVE-23357 > URL: https://issues.apache.org/jira/browse/HIVE-23357 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > There are cases where we may want to submit to a YARN queue as the logged in > user but not require full doas for writing of HDFS or authorization. > A setting could be added to control this behavior: > set hive.server2.tez.queue.access.username.method=default > default is to follow doas setting (the current method) > loggedin is to always username of logged in user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Status: Patch Available (was: In Progress) Want to get an early run through full testing. > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-23354: Attachment: HIVE-23354.1.patch > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
[ https://issues.apache.org/jira/browse/HIVE-23354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23354 started by John Sherman. --- > Remove file size sanity checking from compareTempOrDuplicateFiles > - > > Key: HIVE-23354 > URL: https://issues.apache.org/jira/browse/HIVE-23354 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-23354.1.patch > > > [https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010] > compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to > determine which file(s) to keep. > I've seen instances where this function throws an exception due to the fact > that the newer attemptId file size is less than the older attemptId (thus > failing the query). > I think this assumption is faulty, due to various factors such as file > compression and the order in which values are written. It may be prudent to > trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
