[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[ https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979732#comment-15979732 ] Lefty Leverenz commented on HIVE-16296: --- Doc note: This adds *hive.tez.llap.min.reducer.per.executor* to HiveConf.java, so it needs to be documented in the Tez section of Configuration Properties with a crossreferece in the LLAP section. * [Configuration Properties -- Tez | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Tez] * [Configuration Properties -- LLAP | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP] Added a TODOC3.0 label. > use LLAP executor count to configure reducer auto-parallelism > - > > Key: HIVE-16296 > URL: https://issues.apache.org/jira/browse/HIVE-16296 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch, > HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch, > HIVE-16296.07.patch, HIVE-16296.08.patch, HIVE-16296.09.patch, > HIVE-16296.10.patch, HIVE-16296.10.patch, HIVE-16296.11.patch, > HIVE-16296.12.patch, HIVE-16296.2.patch, HIVE-16296.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[ https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977276#comment-15977276 ] Sergey Shelukhin commented on HIVE-16296: - Not sure why data size estimate changed. It changed by a few bytes; could be how src or the test tables are written; looks like just the usual statistics randomness. > use LLAP executor count to configure reducer auto-parallelism > - > > Key: HIVE-16296 > URL: https://issues.apache.org/jira/browse/HIVE-16296 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch, > HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch, > HIVE-16296.07.patch, HIVE-16296.08.patch, HIVE-16296.09.patch, > HIVE-16296.10.patch, HIVE-16296.10.patch, HIVE-16296.11.patch, > HIVE-16296.12.patch, HIVE-16296.2.patch, HIVE-16296.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976038#comment-15976038
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12864075/HIVE-16296.12.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 10581 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4774/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4774/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4774/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12864075 - PreCommit-HIVE-Build
> use LLAP executor count to configure reducer auto-parallelism
> -
>
> Key: HIVE-16296
> URL: https://issues.apache.org/jira/browse/HIVE-16296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch,
> HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch,
> HIVE-16296.07.patch, HIVE-16296.08.patch, HIVE-16296.09.patch,
> HIVE-16296.10.patch, HIVE-16296.10.patch, HIVE-16296.11.patch,
> HIVE-16296.12.patch, HIVE-16296.2.patch, HIVE-16296.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[ https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975703#comment-15975703 ] Gunther Hagleitner commented on HIVE-16296: --- https://issues.apache.org/jira/secure/attachment/12864075/HIVE-16296.12.patch#file-51 Patch looks good, but I don't understand why the datasize would change? Is this file generated with a different sort order? > use LLAP executor count to configure reducer auto-parallelism > - > > Key: HIVE-16296 > URL: https://issues.apache.org/jira/browse/HIVE-16296 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch, > HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch, > HIVE-16296.07.patch, HIVE-16296.08.patch, HIVE-16296.09.patch, > HIVE-16296.10.patch, HIVE-16296.10.patch, HIVE-16296.11.patch, > HIVE-16296.12.patch, HIVE-16296.2.patch, HIVE-16296.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15974107#comment-15974107
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863937/HIVE-16296.11.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 10563 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[except_distinct]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_all]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_distinct]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[multi_count_distinct_null]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_views]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_views]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cluster]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer1]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cte_1]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lateral_view]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin1]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_column_in]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_types]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_multilevels]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[reduce_deduplicate_extended]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[skiphf_aggr]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_union2]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_coalesce_2]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_complex_all]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_grouping_sets]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_partitioned_date_time]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_distinct_gby]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet]
(batchId=152)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_count_distinct]
(batchId=109)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=236)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4751/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4751/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4751/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973520#comment-15973520
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863871/HIVE-16296.10.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 287 failed/errored test(s), 9370 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=234)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=237)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=237)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=237)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_1]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_view_disable_cbo_1]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_10]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_11]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_12]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_13]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_14]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_1]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_3]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_4]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_5]
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_7]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_add_column2]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrocountemptytbl]
(batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket1] (batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark4]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin10]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin12]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin13]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin5]
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin9]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_input26] (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_gby2_map_multi_distinct]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_union] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_union] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[combine2] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantfolding]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cp_sel] (batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_4] (batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_non_partitioned]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_partitioned]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_rdd_cache]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explode_null]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby1_map]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby1_map_skew]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby1_noskew]
(batchId=31)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971443#comment-15971443
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863666/HIVE-16296.10.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4710/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4710/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4710/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-04-17 18:21:31.926
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4710/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-04-17 18:21:31.930
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
98250bb..8ac7c23 master -> origin/master
c186e2e..99a1226 branch-2 -> origin/branch-2
+ git reset --hard HEAD
HEAD is now at 98250bb HIVE-16197 : Incremental insert into a partitioned table
doesn't get replicated. (Sankar Hariappan via Sushanth Sowmyan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 8ac7c23 HIVE-16287: Alter table partition rename with location -
moves partition back to hive warehouse (Vihang Karajgaonkar, reviewed by Sergio
Pena, Ying Chen)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-04-17 18:21:32.997
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
patching file
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapDecider.java
patching file ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java
patching file ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java
patching file ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java
patching file ql/src/java/org/apache/hadoop/hive/ql/plan/TezEdgeProperty.java
patching file ql/src/test/queries/clientpositive/subquery_scalar.q
patching file ql/src/test/queries/clientpositive/unionDistinct_1.q
patching file ql/src/test/queries/clientpositive/vector_binary_join_groupby.q
patching file
ql/src/test/queries/clientpositive/vector_groupby_grouping_sets_limit.q
patching file ql/src/test/queries/clientpositive/vector_groupby_reduce.q
patching file ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
patching file ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out
patching file
ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out
patching file
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
patching file
ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
patching file
ql/src/test/results/clientpositive/vector_binary_join_groupby.q.out
patching file ql/src/test/results/clientpositive/vector_groupby_reduce.q.out
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958158#comment-15958158
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862170/HIVE-16296.09.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10579 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4569/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4569/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4569/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862170 - PreCommit-HIVE-Build
> use LLAP executor count to configure reducer auto-parallelism
> -
>
> Key: HIVE-16296
> URL: https://issues.apache.org/jira/browse/HIVE-16296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch,
> HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch,
> HIVE-16296.07.patch, HIVE-16296.08.patch, HIVE-16296.09.patch,
> HIVE-16296.2.patch, HIVE-16296.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956084#comment-15956084
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12861950/HIVE-16296.08.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10577 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=234)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=76)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=142)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=220)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4549/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4549/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4549/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12861950 - PreCommit-HIVE-Build
> use LLAP executor count to configure reducer auto-parallelism
> -
>
> Key: HIVE-16296
> URL: https://issues.apache.org/jira/browse/HIVE-16296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch,
> HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch,
> HIVE-16296.07.patch, HIVE-16296.08.patch, HIVE-16296.2.patch, HIVE-16296.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[ https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955768#comment-15955768 ] Sergey Shelukhin commented on HIVE-16296: - So do I, but not HiveQA apparently. > use LLAP executor count to configure reducer auto-parallelism > - > > Key: HIVE-16296 > URL: https://issues.apache.org/jira/browse/HIVE-16296 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch, > HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch, > HIVE-16296.07.patch, HIVE-16296.08.patch, HIVE-16296.2.patch, HIVE-16296.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[ https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955760#comment-15955760 ] Gopal V commented on HIVE-16296: This is a bit strange - I think HDFS has some consistent ordering of files in a directory when listing it. I'm getting consistent results without an ORDER BY (i.e the output files are read in reducer order - reducer 0 files first etc.). > use LLAP executor count to configure reducer auto-parallelism > - > > Key: HIVE-16296 > URL: https://issues.apache.org/jira/browse/HIVE-16296 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch, > HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch, > HIVE-16296.07.patch, HIVE-16296.08.patch, HIVE-16296.2.patch, HIVE-16296.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[ https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954417#comment-15954417 ] Sergey Shelukhin commented on HIVE-16296: - Hmm, results of this test seem to change every run. Will add order by at some point > use LLAP executor count to configure reducer auto-parallelism > - > > Key: HIVE-16296 > URL: https://issues.apache.org/jira/browse/HIVE-16296 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch, > HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch, > HIVE-16296.07.patch, HIVE-16296.2.patch, HIVE-16296.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954411#comment-15954411
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12861763/HIVE-16296.07.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10574 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=234)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=152)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4523/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4523/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4523/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12861763 - PreCommit-HIVE-Build
> use LLAP executor count to configure reducer auto-parallelism
> -
>
> Key: HIVE-16296
> URL: https://issues.apache.org/jira/browse/HIVE-16296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch,
> HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch,
> HIVE-16296.07.patch, HIVE-16296.2.patch, HIVE-16296.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951974#comment-15951974
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12861510/HIVE-16296.06.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10544 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=76)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4501/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4501/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4501/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12861510 - PreCommit-HIVE-Build
> use LLAP executor count to configure reducer auto-parallelism
> -
>
> Key: HIVE-16296
> URL: https://issues.apache.org/jira/browse/HIVE-16296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch,
> HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.06.patch,
> HIVE-16296.2.patch, HIVE-16296.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950056#comment-15950056
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12861317/HIVE-16296.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10540 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit]
(batchId=147)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4473/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4473/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4473/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12861317 - PreCommit-HIVE-Build
> use LLAP executor count to configure reducer auto-parallelism
> -
>
> Key: HIVE-16296
> URL: https://issues.apache.org/jira/browse/HIVE-16296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch,
> HIVE-16296.04.patch, HIVE-16296.05.patch, HIVE-16296.2.patch, HIVE-16296.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944469#comment-15944469
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860766/HIVE-16296.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10510 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit]
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=233)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4404/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4404/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4404/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860766 - PreCommit-HIVE-Build
> use LLAP executor count to configure reducer auto-parallelism
> -
>
> Key: HIVE-16296
> URL: https://issues.apache.org/jira/browse/HIVE-16296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch,
> HIVE-16296.04.patch, HIVE-16296.2.patch, HIVE-16296.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[ https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944282#comment-15944282 ] Sergey Shelukhin commented on HIVE-16296: - Some out files like vector_groupby_grouping_sets_limit, that hive limit w/o gby, also have result changes due to different ordering in absence of order by. > use LLAP executor count to configure reducer auto-parallelism > - > > Key: HIVE-16296 > URL: https://issues.apache.org/jira/browse/HIVE-16296 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch, > HIVE-16296.2.patch, HIVE-16296.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944271#comment-15944271
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860744/HIVE-16296.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 48 failed/errored test(s), 10514 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[except_distinct]
(batchId=138)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_all]
(batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_distinct]
(batchId=138)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[multi_count_distinct_null]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_views]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_views]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cluster]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer1]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cte_1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lateral_view]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin1]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_types]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_multilevels]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[reduce_deduplicate_extended]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[skiphf_aggr]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_union2]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_complex_all]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_grouping_sets]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_partitioned_date_time]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_distinct_gby]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet]
(batchId=150)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection (batchId=232)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid (batchId=232)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg (batchId=232)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[ https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944255#comment-15944255 ] Sergey Shelukhin commented on HIVE-16296: - Looks like a lot of order changes... > use LLAP executor count to configure reducer auto-parallelism > - > > Key: HIVE-16296 > URL: https://issues.apache.org/jira/browse/HIVE-16296 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16296.01.patch, HIVE-16296.03.patch, > HIVE-16296.2.patch, HIVE-16296.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944205#comment-15944205
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860739/HIVE-16296.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 43 failed/errored test(s), 10513 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[except_distinct]
(batchId=138)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_all]
(batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_distinct]
(batchId=138)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[multi_count_distinct_null]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_views]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_views]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cluster]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[correlationoptimizer1]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cte_1]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_4]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lateral_view]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin1]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[parquet_types]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_multilevels]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[reduce_deduplicate_extended]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[skiphf_aggr]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=153)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_union2]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_vector_dynpart_hashjoin_1]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_binary_join_groupby]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_complex_all]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_udf]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_limit]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_grouping_sets]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_partitioned_date_time]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_distinct_gby]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet]
(batchId=150)
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
(batchId=213)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=172)
org.apache.hive.service.server.TestHS2HttpServer.testContextRootUrlRewrite
(batchId=188)
{noformat}
Test results:
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[ https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943745#comment-15943745 ] Gopal V commented on HIVE-16296: The auto-reducer parallelism is entirely disabled for hash-joins. Instead of checking for ARP and bailing, this needs the other ReducerTrait to be passed on (ReducerTraits.UNIFORM). For reduce sinks which have UNIFORM setup, the reducer counts can be increased (but not decreased). ReduceWork can carry a boolean for UNIFORM (which is pretty much the opposite of FIXED) and scale those reducers up as well. > use LLAP executor count to configure reducer auto-parallelism > - > > Key: HIVE-16296 > URL: https://issues.apache.org/jira/browse/HIVE-16296 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16296.01.patch, HIVE-16296.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941696#comment-15941696
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860508/HIVE-16296.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10511 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4361/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4361/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4361/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860508 - PreCommit-HIVE-Build
> use LLAP executor count to configure reducer auto-parallelism
> -
>
> Key: HIVE-16296
> URL: https://issues.apache.org/jira/browse/HIVE-16296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16296.01.patch, HIVE-16296.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16296) use LLAP executor count to configure reducer auto-parallelism
[
https://issues.apache.org/jira/browse/HIVE-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941544#comment-15941544
]
Hive QA commented on HIVE-16296:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860508/HIVE-16296.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10511 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
(batchId=220)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4352/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4352/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4352/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860508 - PreCommit-HIVE-Build
> use LLAP executor count to configure reducer auto-parallelism
> -
>
> Key: HIVE-16296
> URL: https://issues.apache.org/jira/browse/HIVE-16296
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16296.01.patch, HIVE-16296.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
