[jira] [Created] (HIVE-27217) addWriteNotificationLogInBatch can silently fail
John Sherman created HIVE-27217: --- Summary: addWriteNotificationLogInBatch can silently fail Key: HIVE-27217 URL: https://issues.apache.org/jira/browse/HIVE-27217 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: John Sherman Assignee: John Sherman Debugging an issue, I noticed that addWriteNotificationLogInBatch in Hive.java can fail silently if the TApplicationException thrown is not TApplicationException.UNKNOWN_METHOD or TApplicationException.WRONG_METHOD_NAME. https://github.com/apache/hive/blob/40a7d689e51d02fa9b324553fd1810d0ad043080/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3359-L3381 Failures to write in the notification log can be very difficult to debug, we should rethrow the exception so that the failure is very visible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26938) Investigate SMB Map Join for FULL OUTER
John Sherman created HIVE-26938: --- Summary: Investigate SMB Map Join for FULL OUTER Key: HIVE-26938 URL: https://issues.apache.org/jira/browse/HIVE-26938 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: John Sherman Assignee: John Sherman HIVE-18908 added FULL OUTER Map Join support but this work did not add support for SMB Map Joins for FULL OUTER. We should investigate if we can safely support SMB Map Join for this scenario and implement it if so. This is the area in which it gives up conversion, if we modify this line to pass a 2nd argument of true to getBigTableCandidates to enable isFullOuterJoinSupported - it does successfully convert (but we need to verify that execution does the correct thing). [https://github.com/apache/hive/blob/03ad025ada776c0d359124c6342615f1983c1a94/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L482] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26875) Transaction conflict retry loop only executes once
John Sherman created HIVE-26875:
---
Summary: Transaction conflict retry loop only executes once
Key: HIVE-26875
URL: https://issues.apache.org/jira/browse/HIVE-26875
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: John Sherman
Assignee: John Sherman
Currently the "conflict retry loop" only executes once.
[https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L264]
The intent of this loop is to detect if a conflicting transaction has committed
while we were waiting to acquire locks. If there is a conflicting transaction,
it invalidates the snapshot, rolls-back the transaction, opens a new
transaction and tries to re-acquire locks (and then recompile). It then checks
again if a conflicting transaction has committed and if so, redoes the above
steps again, up to HIVE_TXN_MAX_RETRYSNAPSHOT_COUNT times.
However - isValidTxnState relies on getNonSharedLockedTable():
[https://github.com/apache/hive/blob/ab4c53de82d4aaa33706510441167f2df55df15e/ql/src/java/org/apache/hadoop/hive/ql/DriverTxnHandler.java#L422]
which does:
{code:java}
private Set getNonSharedLockedTables() {
if (CollectionUtils.isEmpty(driver.getContext().getHiveLocks())) {
return Collections.emptySet(); // Nothing to check
}{code}
getHiveLocks gets populated by lockAndRespond... HOWEVER -
compileInternal ends up calling compile which ends up calling preparForCompile
which ends up calling prepareContext which ends up destroying the context with
the information lockAndRespond populated. So when the loop executes after all
of this, it will never detect a 2nd conflict because isValidTxnState will
always return true (because it thinks there are no locked objects).
This manifests as duplicate records being created during concurrent UPDATEs if
a transaction get conflicted twice.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26633) Make thrift max message size configurable
John Sherman created HIVE-26633:
---
Summary: Make thrift max message size configurable
Key: HIVE-26633
URL: https://issues.apache.org/jira/browse/HIVE-26633
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 4.0.0-alpha-2
Reporter: John Sherman
Assignee: John Sherman
Since thrift >= 0.14, thrift now enforces max message sizes through a
TConfiguration object as described here:
[https://github.com/apache/thrift/blob/master/doc/specs/thrift-tconfiguration.md]
By default MaxMessageSize gets set to 100MB.
As a result it is possible for HMS clients not to be able to retrieve certain
metadata for tables with a large amount of partitions or other metadata.
For example on a cluster configured with kerberos between hs2 and hms, querying
a large table (10k partitions, 200 columns with names of 200 characters)
results in this backtrace:
{code:java}
org.apache.thrift.transport.TTransportException: MaxMessageSize reached
at
org.apache.thrift.transport.TEndpointTransport.countConsumedMessageBytes(TEndpointTransport.java:96)
at
org.apache.thrift.transport.TMemoryInputTransport.read(TMemoryInputTransport.java:97)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:390)
at
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at
org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:63)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
at
org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:329)
at
org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:273)
at
org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:461)
at
org.apache.hadoop.hive.metastore.api.FieldSchema$FieldSchemaStandardScheme.read(FieldSchema.java:454)
at org.apache.hadoop.hive.metastore.api.FieldSchema.read(FieldSchema.java:388)
at
org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1269)
at
org.apache.hadoop.hive.metastore.api.StorageDescriptor$StorageDescriptorStandardScheme.read(StorageDescriptor.java:1248)
at
org.apache.hadoop.hive.metastore.api.StorageDescriptor.read(StorageDescriptor.java:1110)
at
org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1270)
at
org.apache.hadoop.hive.metastore.api.Partition$PartitionStandardScheme.read(Partition.java:1205)
at org.apache.hadoop.hive.metastore.api.Partition.read(Partition.java:1062)
at
org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:420)
at
org.apache.hadoop.hive.metastore.api.PartitionsByExprResult$PartitionsByExprResultStandardScheme.read(PartitionsByExprResult.java:399)
at
org.apache.hadoop.hive.metastore.api.PartitionsByExprResult.read(PartitionsByExprResult.java:335)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_by_expr_result$get_partitions_by_expr_resultStandardScheme.read(ThriftHiveMetastore.java)
{code}
Making this configurable (and defaulting to a higher value) would allow these
tables to still be accessible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26584) compressed_skip_header_footer_aggr.q is flaky
John Sherman created HIVE-26584:
---
Summary: compressed_skip_header_footer_aggr.q is flaky
Key: HIVE-26584
URL: https://issues.apache.org/jira/browse/HIVE-26584
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 4.0.0-alpha-2
Reporter: John Sherman
Assignee: John Sherman
One of my PRs compressed_skip_header_footer_aggr.q was failing with unexpected
diff. Such as:
{code:java}
TestMiniLlapLocalCliDriver.testCliDriver:62 Client Execution succeeded but
contained differences (error code = 1) after executing
compressed_skip_header_footer_aggr.q
69,71c69,70
< 1 2019-12-31
< 2 2018-12-31
< 3 2017-12-31
---
> 2 2019-12-31
> 3 2019-12-31
89d87
< NULL NULL
91c89
< 2 2018-12-31
---
> 2 2019-12-31
100c98
< 1
---
> 2
109c107
< 1 2019-12-31
---
> 2 2019-12-31
127,128c125,126
< 1 2019-12-31
< 3 2017-12-31
---
> 2 2019-12-31
> 3 2019-12-31
146a145
> 2 2019-12-31
155c154
< 1
---
> 2 {code}
Investigating it, it did not seem to fail when executed locally. Since I
suspected test interference I searched for the tablenames/directories used and
discovered empty_skip_header_footer_aggr.q which uses the same table names AND
external directories.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26574) hive.merge.tezfiles does not work with LLAP hive.llap.execution.mode = all
John Sherman created HIVE-26574: --- Summary: hive.merge.tezfiles does not work with LLAP hive.llap.execution.mode = all Key: HIVE-26574 URL: https://issues.apache.org/jira/browse/HIVE-26574 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 4.0.0-alpha-2 Reporter: John Sherman Assignee: John Sherman set hive.execution.mode=llap; set hive.llap.execution.mode=all; set hive.merge.tezfiles=true; with the above settings the Merge work generated does not get converted to LLAP work and thus fails on platform that require LLAP. This will either end up causing the query compilation to fail OR a NullPointerException at execution time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26472) Concurrent UPDATEs can cause duplicate rows
John Sherman created HIVE-26472:
---
Summary: Concurrent UPDATEs can cause duplicate rows
Key: HIVE-26472
URL: https://issues.apache.org/jira/browse/HIVE-26472
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 4.0.0-alpha-1
Reporter: John Sherman
Attachments: debug.diff
Concurrent UPDATEs to the same table can cause duplicate rows when the
following occurs:
Two UPDATEs get assigned txnIds and writeIds like this:
UPDATE #1 = txnId: 100 writeId: 50 <--- commits first
UPDATE #2 = txnId: 101 writeId: 49
To replicate the issue:
I applied the attach debug.diff patch which adds hive.lock.sleep.writeid (which
controls the amount to sleep before acquiring a writeId) and
hive.lock.sleep.post.writeid (which controls the amount to sleep after
acquiring a writeId).
{code:java}
CREATE TABLE test_update(i int) STORED AS ORC
TBLPROPERTIES('transactional'="true");
INSERT INTO test_update VALUES (1);
Start two beeline connections.
In connection #1 - run:
set hive.driver.parallel.compilation = true;
set hive.lock.sleep.writeid=5s;
update test_update set i = 1 where i = 1;
Wait one second and in connection #2 - run:
set hive.driver.parallel.compilation = true;
set hive.lock.sleep.post.writeid=10s;
update test_update set i = 1 where i = 1;
After both updates complete - it is likely that test_update contains two rows
now.
{code}
HIVE-24211 seems to address the case when:
UPDATE #1 = txnId: 100 writeId: 50
UPDATE #2 = txnId: 101 writeId: 49 <--- commits first (I think this causes
UPDATE #1 to detect the snapshot is out of date because commitedTxn > UPDATE
#1s txnId)
A possible work around is to set hive.driver.parallel.compilation = false, but
this would only help in cases there is only one HS2 instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26423) Make commons-pool2 a explicit dependency
John Sherman created HIVE-26423: --- Summary: Make commons-pool2 a explicit dependency Key: HIVE-26423 URL: https://issues.apache.org/jira/browse/HIVE-26423 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: John Sherman Assignee: John Sherman HIVE-26242 started using the commons-pool2 which is getting pulled in a transitive dependency through commons-dbcp2 or calcite-core. It would be better to make it an explicit dependency to ensure it gets packaged properly and/or things do not suddenly break if the transitive dependencies change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26418) Stop packaging hive-standalone-metastore-server*-tests.jar
John Sherman created HIVE-26418:
---
Summary: Stop packaging hive-standalone-metastore-server*-tests.jar
Key: HIVE-26418
URL: https://issues.apache.org/jira/browse/HIVE-26418
Project: Hive
Issue Type: Bug
Reporter: John Sherman
Assignee: John Sherman
HIVE-25989 added hive-standalone-metastore-server with the classifier of tests
to hbase-handler/pom.xml but without the scope of test.As a result
hive-standalone-metastore-server*-tests.jar gets packaged.
{code:java}
(packaging) $ find . -iname "*test*.jar"
./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-llap-common-4.0.0-alpha-2-SNAPSHOT-tests.jar
./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-testutils-4.0.0-alpha-2-SNAPSHOT.jar
./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-standalone-metastore-server-4.0.0-alpha-2-SNAPSHOT-tests.jar
./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/opentest4j-1.2.0.jar
{code}
Changing the pom file to use scope test for the tests jar and to also bring in
the non-tests jar for runtime dependency fixes the issue:
{code:java}
(packaging) $ find . -iname "*test*.jar"
./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-llap-common-4.0.0-alpha-2-SNAPSHOT-tests.jar
./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/hive-testutils-4.0.0-alpha-2-SNAPSHOT.jar
./target/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/apache-hive-4.0.0-alpha-2-SNAPSHOT-bin/lib/opentest4j-1.2.0.jar
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26398) INSERT INTO complex type with varchar/char field fails CBO
John Sherman created HIVE-26398:
---
Summary: INSERT INTO complex type with varchar/char field fails CBO
Key: HIVE-26398
URL: https://issues.apache.org/jira/browse/HIVE-26398
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 4.0.0
Reporter: John Sherman
Assignee: John Sherman
{code}
CREATE EXTERNAL TABLE test_complex_struct(struct_type struct) STORED AS PARQUET;
INSERT INTO test_complex_struct SELECT named_struct("string_type", 'str1',
"varchar_type", cast('varchar1' as varchar(100)), "char_type", cast('char' as
char(34)));
{code}
fails with
{code}
org.apache.hadoop.hive.ql.parse.SemanticException: Line 2:12 Cannot insert
into target table because column number/types are different
'test_complex_struct': Cannot convert column 0 from
struct to
struct.{code}
This also fails with maps:
{code}
CREATE EXTERNAL TABLE test_complex_map(t_map map)
STORED AS PARQUET;
INSERT INTO test_complex_map SELECT map(cast("test" as varchar(100)),
cast("value" as char(10)));
{code}
It seems that the cast is being lost or ignored somewhere along the way (and it
stays a string).
The queries do complete when ran on the non-CBO path. (Disabling CBO or falling
back)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26395) Support CREATE TABLE LIKE FILE for PARQUET
John Sherman created HIVE-26395:
---
Summary: Support CREATE TABLE LIKE FILE for PARQUET
Key: HIVE-26395
URL: https://issues.apache.org/jira/browse/HIVE-26395
Project: Hive
Issue Type: New Feature
Components: HiveServer2
Reporter: John Sherman
Assignee: John Sherman
The intent is to allow a user to create a table and derive the schema from a
user provided parquet file. A secondary goal is to generalize this support so
other SerDes/formats could implement the feature easily.
The proposed syntax is:
CREATE TABLE LIKE FILE 'path to file';
Example being:
{code:java}
CREATE TABLE like_test_all_types LIKE FILE PARQUET
'${system:test.tmp.dir}/test_all_types/00_0';{code}
with partitioning
{code}
CREATE TABLE like_test_partitioning LIKE FILE PARQUET
'${system:test.tmp.dir}/test_all_types/00_0' PARTITIONED BY (year STRING,
month STRING);
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26038) Respect Owner privileges in all paths in AuthorizationMetaStoreFilterHook
John Sherman created HIVE-26038:
---
Summary: Respect Owner privileges in all paths in
AuthorizationMetaStoreFilterHook
Key: HIVE-26038
URL: https://issues.apache.org/jira/browse/HIVE-26038
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: John Sherman
Currently there are some paths in AuthorizationMetaStoreFilterHook that do not
provide object ownership information to the authorization plugin. Specifically
- any method using filterDatabases or filterTableNames will not include object
ownership in the privilege request. This is problematic when an application
using Hive uses these code paths together:
Example:
Applications creates a table, inserts into, selects from it through HS2.
Explicit privilege is never given to the table in the authorization provider -
it is relying on the fact that the user is owner of the table. (The user has
CREATE privilege on the DB in question to be able to CREATE the table).
Application then uses:
service/src/java/org/apache/hive/service/cli/operation/GetTablesOperation.java
to get a list of tables. Which in turn calls metastoreClient.getTableMeta -
which ends up using filterTableNames to remove table entries that the user is
not authorized to see. But filterTableNames does not provide table ownership
information to the authorization framework so the framework will filter out
tables user actually has ownership of.
There are two methods here:
{code:java}
public List filterTableNames(String catName, String dbName,
List tableList){code}
vs
{code:java}
public List filterTables(List tableList) throws
MetaException{code}
First one only takes in a list of strings and would be impossible to provide
proper ownership information (without additional HMS calls) - we should remove
this method and modify all code paths to use filterTables. This would involve
some plumbing and adding additional information to various HMS responses to
include ownership information.
The same problem exists for the database code paths.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HIVE-25768) Extended query-level HMS cache lifetime beyond analysis stage
John Sherman created HIVE-25768: --- Summary: Extended query-level HMS cache lifetime beyond analysis stage Key: HIVE-25768 URL: https://issues.apache.org/jira/browse/HIVE-25768 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: John Sherman Assignee: John Sherman HIVE-24176 added a HMS response cache to improve compilation times when metadata is requested multiple times. The cache gets destroyed after analysis. If we extend the lifetime of the cache beyond analysis it could be used to reduce HMS communication for other areas such as various exec hooks that inspect the metadata for a query. The impetus that motivated this change is specifically the Atlas hook which generates lineage information and makes HMS calls. These calls to HMS can add latency to query results. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HIVE-24645) UDF configure not called when fetch task conversion occurs
John Sherman created HIVE-24645: --- Summary: UDF configure not called when fetch task conversion occurs Key: HIVE-24645 URL: https://issues.apache.org/jira/browse/HIVE-24645 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: John Sherman Assignee: John Sherman When hive.fetch.task.conversion kicks in - UDF configure is not called. This is likely due to MapredContext not being available when this conversion occurs. The approach I suggest is to create a dummy MapredContext and provide it with the current configuration from ExprNodeGenericFuncEvaluator. It is slightly unfortunate that the UDF API relies on MapredContext since some aspects of the context do not apply to the variety of engines and invocation paths for UDFs which makes it difficult to make a fully formed dummy object such as the Reporter objects and the boolean around if it is a Map context. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24556) Optimize DefaultGraphWalker for case when node has no grandchildren
John Sherman created HIVE-24556: --- Summary: Optimize DefaultGraphWalker for case when node has no grandchildren Key: HIVE-24556 URL: https://issues.apache.org/jira/browse/HIVE-24556 Project: Hive Issue Type: Improvement Affects Versions: 4.0.0 Reporter: John Sherman Assignee: John Sherman Investigating query with large IN clause with constant strings (100k+) taking significant time during compilation revealed a possible optimization within DefaultGraphWalker. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24550) Cleanup only transaction information for the current DriverContext
John Sherman created HIVE-24550: --- Summary: Cleanup only transaction information for the current DriverContext Key: HIVE-24550 URL: https://issues.apache.org/jira/browse/HIVE-24550 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 4.0.0 Reporter: John Sherman Assignee: John Sherman Long term solution would be: https://issues.apache.org/jira/browse/HIVE-24549 Short term solution for the common usage pattern described in HIVE-24549 is to ensure the current driverContext queryId matches the TxnManagers queryId. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24549) TxnManager should not be shared across queries
John Sherman created HIVE-24549:
---
Summary: TxnManager should not be shared across queries
Key: HIVE-24549
URL: https://issues.apache.org/jira/browse/HIVE-24549
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 4.0.0
Reporter: John Sherman
Assignee: John Sherman
There are various sections of code that assume the DbTxnManager is not shared
across concurrent queries in a session.
Such as (which gets invoked during closeOperation):
[https://github.com/apache/hive/blob/3f5e01cae5b65dde7edb3fbde8ebe70c1d02f6cf/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L868-L885]
{code:java}
// is usually called after close() to commit or rollback a query and end the
driver life cycle.
// do not understand why it is needed and wonder if it could be combined with
close.
@Override
public void destroy() {
driverState.lock();
try {
// in the cancel case where the driver state is INTERRUPTED, destroy will
be deferred to
// the query process
if (driverState.isDestroyed()) {
return;
} else {
driverState.descroyed();
}
} finally {
driverState.unlock();
}
driverTxnHandler.destroy();
}
{code}
The problematic part is the: driverTxnHandler.destroy() which looks like:
{code:java}
void destroy() {
boolean isTxnOpen =
driverContext != null &&
driverContext.getTxnManager() != null &&
driverContext.getTxnManager().isTxnOpen();
release(!hiveLocks.isEmpty() || isTxnOpen);
}
{code}
What happens is (rough sketch):
Q1 - starts operation, acquires txn, does operation, closes txn/cleans up txn
info, starts fetching data
Q2 - starts operation, acquire txn
Q1 - calls close operation which in turn calls destroy which sees the Q2s
transaction information and cleans it up.
Q2 - proceeds and fails in splitGeneration when it no longer can find its
Valid*TxnIdList information.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-23699) Cleanup HIVEQUERYRESULTFILEFORMAT handling
John Sherman created HIVE-23699: --- Summary: Cleanup HIVEQUERYRESULTFILEFORMAT handling Key: HIVE-23699 URL: https://issues.apache.org/jira/browse/HIVE-23699 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 4.0.0 Reporter: John Sherman Assignee: John Sherman HIVEQUERYRESULTFILEFORMAT format handling has grown over the years and has become somewhat messy code wise in SemanticAnalyzer and TaskCompiler. There are special cases where the HIVEQUERYRESULTFILEFORMAT setting gets changed at runtime that may cause issues if a user changes execution.engines between queries and probably other corner cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23614) Always pass HiveConfig to removeTempOrDuplicateFiles
John Sherman created HIVE-23614: --- Summary: Always pass HiveConfig to removeTempOrDuplicateFiles Key: HIVE-23614 URL: https://issues.apache.org/jira/browse/HIVE-23614 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: John Sherman Assignee: John Sherman As part of HIVE-23354, we check the provided HiveConf for speculative execution and throw an error if it is enabled. There is one path that did not previously provide a HiveConf value which shows up in test failures in runtime_skewjoin_mapjoin_spark.q, skewjoin.q and skewjoin_onesideskew.q (which we do not run as part of pre-commit tests). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23478) Fix flaky special_character_in_tabnames_quotes_1 test
John Sherman created HIVE-23478: --- Summary: Fix flaky special_character_in_tabnames_quotes_1 test Key: HIVE-23478 URL: https://issues.apache.org/jira/browse/HIVE-23478 Project: Hive Issue Type: Improvement Components: Tests Affects Versions: 4.0.0 Reporter: John Sherman Assignee: John Sherman Fix For: 4.0.0 While testing https://issues.apache.org/jira/browse/HIVE-23354 special_character_in_tabnames_quotes_1 failed. Searching for the test, it seems other patches have also had failures. I noticed that special_character_in_tabnames_1 and special_character_in_tabnames_quotes_1 use the same database/table names. I suspect this is responsible for some of the flakiness. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23357) Allow queue user to be configured separately from doAs
John Sherman created HIVE-23357: --- Summary: Allow queue user to be configured separately from doAs Key: HIVE-23357 URL: https://issues.apache.org/jira/browse/HIVE-23357 Project: Hive Issue Type: Improvement Components: HiveServer2 Affects Versions: 4.0.0 Reporter: John Sherman Assignee: John Sherman There are cases where we may want to submit to a YARN queue as the logged in user but not require full doas for writing of HDFS or authorization. A setting could be added to control this behavior: set hive.server2.tez.queue.access.username.method=default default is to follow doas setting (the current method) loggedin is to always username of logged in user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23355) Handle STORED AS quoting behavior properly when masking is enabled
John Sherman created HIVE-23355:
---
Summary: Handle STORED AS quoting behavior properly when masking
is enabled
Key: HIVE-23355
URL: https://issues.apache.org/jira/browse/HIVE-23355
Project: Hive
Issue Type: Bug
Components: HiveServer2
Reporter: John Sherman
Assignee: John Sherman
Fix For: 4.0.0
Test framework repro:
{code:java}
--- a/ql/src/test/queries/clientpositive/masking_reserved.q
+++ b/ql/src/test/queries/clientpositive/masking_reserved.q
@@ -5,6 +5,7 @@ set
hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.autho
create table keyword_test_off (id int, `etad` string, key int);
create table keyword_test_on (id int, `date` string, key int);
create table masking_test_n_masking_reserved (id int, value string, key int);
+create temporary table masking_test_n_masking_temp stored as orc as select *
from masking_test_n_masking_reserved; explain select a.`etad`, b.value from
keyword_test_off a join masking_test_n_masking_reserved b on b.id = a.id;
select a.`etad`, b.value from keyword_test_off a join
masking_test_n_masking_reserved b on b.id = a.id; {code}
fails with:
{code:java}
2020-04-30T20:19:06,833 INFO [bba048b2-380a-46e2-b283-52c1090a0d5c main]
parse.CalcitePlanner: Creating table default.masking_test_n_masking_temp
position=23
2020-04-30T20:19:06,851 ERROR [bba048b2-380a-46e2-b283-52c1090a0d5c main]
ql.Driver: FAILED: SemanticException Unrecognized file format in STORED AS
clause: '`ORC`'
org.apache.hadoop.hive.ql.parse.SemanticException: Unrecognized file format in
STORED AS clause: '`ORC`'
at
org.apache.hadoop.hive.ql.parse.StorageFormat.processStorageFormat(StorageFormat.java:89)
at
org.apache.hadoop.hive.ql.parse.StorageFormat.fillStorageFormat(StorageFormat.java:74)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13351)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12256)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12471)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:432)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:184) {code}
We should either not quote the STORED BY argument in this path OR strip quotes
in processStorageFormat.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-23354) Remove file size sanity checking from compareTempOrDuplicateFiles
John Sherman created HIVE-23354: --- Summary: Remove file size sanity checking from compareTempOrDuplicateFiles Key: HIVE-23354 URL: https://issues.apache.org/jira/browse/HIVE-23354 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: John Sherman Assignee: John Sherman https://github.com/apache/hive/blob/cdd55aa319a3440963a886ebfff11cd2a240781d/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L1952-L2010 compareTempOrDuplicateFiles uses a combination of attemptId and fileSize to determine which file(s) to keep. I've seen instances where this function throws an exception due to the fact that the newer attemptId file size is less than the older attemptId (thus failing the query). I think this assumption is faulty, due to various factors such as file compression and the order in which values are written. It may be prudent to trust that the newest attemptId is in fact the best choice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23106) Cleanup CalcitePlanner genOPTree exception handling
John Sherman created HIVE-23106: --- Summary: Cleanup CalcitePlanner genOPTree exception handling Key: HIVE-23106 URL: https://issues.apache.org/jira/browse/HIVE-23106 Project: Hive Issue Type: Improvement Components: HiveServer2 Reporter: John Sherman Assignee: John Sherman The logic where genOPTree handles exceptions during CBO planning is a bit twisty and could use some cleanup and comments. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22594) Reorder metastore thrift structures
John Sherman created HIVE-22594: --- Summary: Reorder metastore thrift structures Key: HIVE-22594 URL: https://issues.apache.org/jira/browse/HIVE-22594 Project: Hive Issue Type: Improvement Components: Standalone Metastore, Thrift API Affects Versions: 4.0.0 Reporter: John Sherman Assignee: John Sherman Currently hive_metastore.thrift C++ output does not compile due to structures like ScheduledQueryKey and ColumnStatistics being referenced before being defined later in the file. We should reorder the structures in the file so the C++ output is usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22351) Fix incorrect threaded ObjectStore usage in TestObjectStore
John Sherman created HIVE-22351: --- Summary: Fix incorrect threaded ObjectStore usage in TestObjectStore Key: HIVE-22351 URL: https://issues.apache.org/jira/browse/HIVE-22351 Project: Hive Issue Type: Bug Affects Versions: 4.0.0 Reporter: John Sherman Assignee: John Sherman Fix For: 4.0.0 TestObjectStore#testConcurrentDropPartitions shares the same ObjectStore object across two threads doing concurrent drops. ObjectStore isn't thread safe. I've seen occasional deadlocks where one thread is rolling back and another is trying to commit with each thread waiting on the other (one is holding an object lock via a synchronized method and another is holding the ExecutionContextThreadedImpl lock). I think the intent is to test dropPartition behavior and ensure the backing DB receives a rollback or abort on conflicting drops. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22289) Regenerate test output for tests broken due to commit race
John Sherman created HIVE-22289: --- Summary: Regenerate test output for tests broken due to commit race Key: HIVE-22289 URL: https://issues.apache.org/jira/browse/HIVE-22289 Project: Hive Issue Type: Task Reporter: John Sherman Assignee: John Sherman HIVE-22042 got committed which changed the plans of a few tests (by enabling nonstrict partitioning mode by default) then HIVE-22269 got committed which fixes a bug with stats not being correctly calculated on some operators. Each patch got green runs individually but together causes test output differences. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22286) Disable flaky TestBuddyAllocator#testMTT test
John Sherman created HIVE-22286: --- Summary: Disable flaky TestBuddyAllocator#testMTT test Key: HIVE-22286 URL: https://issues.apache.org/jira/browse/HIVE-22286 Project: Hive Issue Type: Task Affects Versions: 4.0.0 Reporter: John Sherman Assignee: John Sherman Fix For: 4.0.0 TestBuddyAllocator#TestMTT has been flaky for awhile, failing 20-30% of the time. Disabling the test until HIVE-22175 is reviewed and committed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
John Sherman created HIVE-22252: --- Summary: Fix caught NullPointerExceptions generated during EXPLAIN Key: HIVE-22252 URL: https://issues.apache.org/jira/browse/HIVE-22252 Project: Hive Issue Type: Bug Reporter: John Sherman Assignee: John Sherman While debugging an issue I noticed that during EXPLAIN the following methods throw a NullPointerException: VectorColumnOutputMapping::finalize AbstractOperatorDesc::getUserLevelStatistics AbstractOperatorDesc::getColumnExprMapForExplain The exceptions do end up getting caught but we should add null checks and gracefully to be less wasteful and to aid future debugging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
