[
https://issues.apache.org/jira/browse/HIVE-22002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903000#comment-16903000
]
Hive QA commented on HIVE-22002:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12976977/HIVE-22002.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 16723 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[analyze_table_null_partition]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[drop_partitions_filter4]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_coltype_literals]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partitions_filter_default]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_join_partition_key]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_only_null]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_non_string_partition]
(batchId=37)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_partitioned]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_only_null]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_non_string_partition]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[dynpart_sort_optimization]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats_only_null]
(batchId=125)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testExceptions (batchId=350)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testJoin (batchId=350)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testReadWrite (batchId=350)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadMultiPartition
(batchId=350)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadPartition
(batchId=350)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWritePartition
(batchId=350)
org.apache.hive.streaming.TestStreamingDynamicPartitioning.testDPTwoLevelNonStringPartitionColumns
(batchId=357)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18280/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18280/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18280/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12976977 - PreCommit-HIVE-Build
> Insert into table partition fails partially with stats.autogather is on.
> ------------------------------------------------------------------------
>
> Key: HIVE-22002
> URL: https://issues.apache.org/jira/browse/HIVE-22002
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 4.0.0
> Reporter: Naveen Gangam
> Assignee: bencao
> Priority: Major
> Attachments: HIVE-22002.patch, image-2019-07-31-20-02-38-069.png
>
>
> create table test_double(id int) partitioned by (dbtest double);
> insert into test_double partition(dbtest) values (1,9.9); --> this works
> insert into test_double partition(dbtest) values (1,10); --> this fails
> But if we change it to
> insert into test_double partition(dbtest) values (1, cast (10 as double)); it
> succeeds
> -> the problem is only seen when trying to insert a whole number i.e. 10,
> 10.0, 15, 14.0 etc. The issue is not seen when inserting a number with
> decimal values other than 0. So insert of 10.1 goes though.
> The underlying from the HMS is
> {code}
> 2019-07-11T07:58:16,670 [pool-6-thread-196]: server.TThreadPoolServer
> (TThreadPoolServer.java:run(297)) - occurred during processing of message.
> java.lang.IndexOutOfBoundsException: Index: 0 at
> java.util.Collections$EmptyList.get(Collections.java:4454) ~[?:1.8.0_112] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:7808)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78] at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7769)
> ~[hive-exec-3.1.0.3.1.0.0-78.jar:3.1.0.3.1.0.0-78]
> {code}
> With {{hive.stats.column.autogather=false}}, this exception does not occur
> with or without the explicit casting.
> The issue stems from the fact that HS2 created a partition with value
> {{dbtest=10}} for the table and the stats processor is attempting to add
> column statistics for partition with value {{dbtest=10.0}}. Thus HMS
> {{getPartitionsByNames}} cannot find the partition with that value and thus
> fails to insert the stats. So while the failure initiates on HMS side, the
> cause in the HS2 query planning.
> It makes sense that turning off {{hive.stats.column.autogather}} resolves the
> issue because there is no StatsTask in a query plan.
> But {{SHOW PARTITIONS}} shows the partition as created while the query
> planner is not including it any plan because of the absence of stats on the
> partition.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
