[
https://issues.apache.org/jira/browse/HIVE-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250477#comment-16250477
]
Hive QA commented on HIVE-17935:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12897347/HIVE-17935.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 11380 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ppd_union_view]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_acid_part_update]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_all_partitioned]
(batchId=160)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnCommands2.testDynamicPartitionsMerge2
(batchId=274)
org.apache.hadoop.hive.ql.TestTxnCommands2.testMultiInsert (batchId=274)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testDynamicPartitionsMerge2
(batchId=284)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMultiInsert
(batchId=284)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.schemaEvolutionAddColDynamicPartitioningUpdate
(batchId=221)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=233)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=233)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=230)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7793/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7793/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7793/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12897347 - PreCommit-HIVE-Build
> Turn on hive.optimize.sort.dynamic.partition by default
> -------------------------------------------------------
>
> Key: HIVE-17935
> URL: https://issues.apache.org/jira/browse/HIVE-17935
> Project: Hive
> Issue Type: Bug
> Reporter: Andrew Sherman
> Assignee: Andrew Sherman
> Attachments: HIVE-17935.1.patch, HIVE-17935.2.patch,
> HIVE-17935.3.patch, HIVE-17935.4.patch, HIVE-17935.5.patch
>
>
> The config option hive.optimize.sort.dynamic.partition is an optimization for
> Hive’s dynamic partitioning feature. It was originally implemented in
> [HIVE-6455|https://issues.apache.org/jira/browse/HIVE-6455]. With this
> optimization, the dynamic partition columns and bucketing columns (in case of
> bucketed tables) are sorted before being fed to the reducers. Since the
> partitioning and bucketing columns are sorted, each reducer can keep only one
> record writer open at any time thereby reducing the memory pressure on the
> reducers. There were some early problems with this optimization and it was
> disabled by default in HiveConf in
> [HIVE-8151|https://issues.apache.org/jira/browse/HIVE-8151]. Since then
> setting hive.optimize.sort.dynamic.partition=true has been used to solve
> problems where dynamic partitioning produces with (1) too many small files on
> HDFS, which is bad for the cluster and can increase overhead for future Hive
> queries over those partitions, and (2) OOM issues in the map tasks because it
> trying to simultaneously write to 100 different files.
> It now seems that the feature is probably mature enough that it can be
> enabled by default.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
