[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794327#comment-16794327
]
Gopal V commented on HIVE-21399:
{code}
ERROR: -1 due to no test(s) being added or modified.
{code}
I don't believe that warning anymore.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch,
> HIVE-21399.03.patch, HIVE-21399.04.patch, HIVE-21399.05.patch,
> HIVE-21399.06.patch, HIVE-21399.07.patch, HIVE-21399.08.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794094#comment-16794094
]
Hive QA commented on HIVE-21399:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962668/HIVE-21399.08.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15833 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16536/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16536/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16536/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962668 - PreCommit-HIVE-Build
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch,
> HIVE-21399.03.patch, HIVE-21399.04.patch, HIVE-21399.05.patch,
> HIVE-21399.06.patch, HIVE-21399.07.patch, HIVE-21399.08.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794085#comment-16794085
]
Hive QA commented on HIVE-21399:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2257 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
55s{color} | {color:red} ql: The patch generated 5 new + 700 unchanged - 2
fixed = 705 total (was 702) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16536/dev-support/hive-personality.sh
|
| git revision | master / 6a33d26 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16536/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql accumulo-handler contrib hbase-handler
itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16536/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch,
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794058#comment-16794058
]
Gopal V commented on HIVE-21399:
The patch LGTM - +1
A significant number of them are 0.0 or close to
{code}
$ grep "minRed.*:" HIVE-21399.08.patch | grep -v 0.99 | sed "s/^+ *//" |sort |
uniq -c
2198 minReductionHashAggr: 0.0
2 minReductionHashAggr: 0.0077519417
3 minReductionHashAggr: 0.05582
10 minReductionHashAggr: 0.1669
1 minReductionHashAggr: 0.278
34 minReductionHashAggr: 0.333
1 minReductionHashAggr: 0.46745563
5 minReductionHashAggr: 0.5
1 minReductionHashAggr: 0.536193
1 minReductionHashAggr: 0.5666
1 minReductionHashAggr: 0.5879121
4 minReductionHashAggr: 0.666
1 minReductionHashAggr: 0.76018816
{code}
I'll look through all the cases in some more detail - for example, the
{{0.0077519417}} value seems to be related to a compute-stats query.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch,
> HIVE-21399.03.patch, HIVE-21399.04.patch, HIVE-21399.05.patch,
> HIVE-21399.06.patch, HIVE-21399.07.patch, HIVE-21399.08.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[ https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794053#comment-16794053 ] Jesus Camacho Rodriguez commented on HIVE-21399: [~ashutoshc], [~gopalv], could you take a look? Thanks > Adjust hive.map.aggr.hash.min.reduction statically depending on group by > statistics > --- > > Key: HIVE-21399 > URL: https://issues.apache.org/jira/browse/HIVE-21399 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch, > HIVE-21399.03.patch, HIVE-21399.04.patch, HIVE-21399.05.patch, > HIVE-21399.06.patch, HIVE-21399.07.patch, HIVE-21399.08.patch > > > Currently, the value is set statically from config variable. If stats are > available, we could try to adjust this value at optimization time to favor > turning off hash aggregation earlier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793297#comment-16793297
]
Hive QA commented on HIVE-21399:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962508/HIVE-21399.07.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15827 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_join]
(batchId=86)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_join]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_6]
(batchId=191)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16518/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16518/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16518/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962508 - PreCommit-HIVE-Build
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch,
> HIVE-21399.03.patch, HIVE-21399.04.patch, HIVE-21399.05.patch,
> HIVE-21399.06.patch, HIVE-21399.07.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793292#comment-16793292
]
Hive QA commented on HIVE-21399:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
19s{color} | {color:blue} ql in master has 2257 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
53s{color} | {color:red} ql: The patch generated 5 new + 700 unchanged - 2
fixed = 705 total (was 702) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 21 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16518/dev-support/hive-personality.sh
|
| git revision | master / 3f02ce4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16518/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16518/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16518/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql accumulo-handler contrib hbase-handler
itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16518/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL:
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792475#comment-16792475
]
Hive QA commented on HIVE-21399:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962407/HIVE-21399.06.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16502/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16502/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16502/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12962407/HIVE-21399.06.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962407 - PreCommit-HIVE-Build
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch,
> HIVE-21399.03.patch, HIVE-21399.04.patch, HIVE-21399.05.patch,
> HIVE-21399.06.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792322#comment-16792322
]
Hive QA commented on HIVE-21399:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962407/HIVE-21399.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15825 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_logical]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_merge_multi_expressions]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_9]
(batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part4]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part8]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_3] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_disablecbo_3]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[test_teradatabinaryfile]
(batchId=2)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16497/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16497/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16497/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962407 - PreCommit-HIVE-Build
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch,
> HIVE-21399.03.patch, HIVE-21399.04.patch, HIVE-21399.05.patch,
> HIVE-21399.06.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792313#comment-16792313
]
Hive QA commented on HIVE-21399:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2257 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
53s{color} | {color:red} ql: The patch generated 5 new + 700 unchanged - 2
fixed = 705 total (was 702) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16497/dev-support/hive-personality.sh
|
| git revision | master / 05ca910 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16497/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql accumulo-handler contrib hbase-handler
itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16497/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch,
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791423#comment-16791423
]
Hive QA commented on HIVE-21399:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962252/HIVE-21399.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 15787 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=230)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=230)
TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely
timed out) (batchId=230)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[binary_output_format]
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_dp]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cp_sel] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_logical]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part2] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_merge_multi_expressions]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_9]
(batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part4]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part8]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_3] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_disablecbo_3]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[rand_partitionpruner2]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats4] (batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_25]
(batchId=96)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_partition_pruning_2]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_semijoin_user_level]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dpp]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_groupingset_bug]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_partitioned]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part1]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_dyn_part2]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_no_join_opt_2]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_part_1]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_part_2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[reopt_dpp]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_input_counters]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_part]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_part_varchar]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_join_part_col_char]
(batchId=109)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16480/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16480/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16480/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 39 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791406#comment-16791406
]
Hive QA commented on HIVE-21399:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
53s{color} | {color:red} ql: The patch generated 5 new + 700 unchanged - 2
fixed = 705 total (was 702) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16480/dev-support/hive-personality.sh
|
| git revision | master / 13938db |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16480/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql accumulo-handler contrib hbase-handler
itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16480/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch,
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787650#comment-16787650
]
Hive QA commented on HIVE-21399:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961628/HIVE-21399.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1400 failed/errored test(s), 15820 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=267)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=267)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[explain_outputs]
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_16]
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=275)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[explain]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=278)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats3] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats4] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStatsPart]
(batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_deep_filters]
(batchId=98)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby2]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_11]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5a]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_6]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join0] (batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join10] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join11] (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join12] (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join13] (batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join14] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join15] (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join16] (batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join17] (batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join18] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join18_multi_distinct]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19_inclause]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join1] (batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join20] (batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join22] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join24] (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join26] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join27] (batchId=97)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join2] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join31] (batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join32] (batchId=93)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787625#comment-16787625
]
Hive QA commented on HIVE-21399:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 5 new + 700 unchanged - 2
fixed = 705 total (was 702) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16402/dev-support/hive-personality.sh
|
| git revision | master / bd95a2f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16402/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16402/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch,
> HIVE-21399.03.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786762#comment-16786762
]
Hive QA commented on HIVE-21399:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961488/HIVE-21399.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16384/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16384/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16384/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12961488/HIVE-21399.02.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961488 - PreCommit-HIVE-Build
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786688#comment-16786688
]
Hive QA commented on HIVE-21399:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961488/HIVE-21399.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16382/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16382/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16382/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12961488/HIVE-21399.02.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12961488 - PreCommit-HIVE-Build
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786687#comment-16786687
]
Hive QA commented on HIVE-21399:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12961488/HIVE-21399.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1413 failed/errored test(s), 15819 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=267)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=267)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[explain_outputs]
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_16]
(batchId=275)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=275)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[explain]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=278)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[write_final_output_blobstore]
(batchId=278)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats3] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats4] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alterColumnStatsPart]
(batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_deep_filters]
(batchId=98)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby2]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_11]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5a]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_6]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join0] (batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join10] (batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join11] (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join12] (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join13] (batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join14] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join15] (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join16] (batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join17] (batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join18] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join18_multi_distinct]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join19_inclause]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join1] (batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join20] (batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join22] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join24] (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join26] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join27] (batchId=97)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join2] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join31] (batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join32] (batchId=93)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786659#comment-16786659
]
Hive QA commented on HIVE-21399:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
30s{color} | {color:blue} ql in master has 2251 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 5 new + 700 unchanged - 2
fixed = 705 total (was 702) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16381/dev-support/hive-personality.sh
|
| git revision | master / 84f766e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16381/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16381/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[ https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786359#comment-16786359 ] Gopal V commented on HIVE-21399: That might explain why it isn't working for me right now - the config is very optimistic and the stats say be pessimistic. My config is 0.99 and I'm testing to see if a massive nDV product reduces it and skips map aggregation faster. > Adjust hive.map.aggr.hash.min.reduction statically depending on group by > statistics > --- > > Key: HIVE-21399 > URL: https://issues.apache.org/jira/browse/HIVE-21399 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch > > > Currently, the value is set statically from config variable. If stats are > available, we could try to adjust this value at optimization time to favor > turning off hash aggregation earlier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786357#comment-16786357
]
Jesus Camacho Rodriguez commented on HIVE-21399:
[~gopalv], new value is calculated as: {{max(config_value, 1 - ndv/num_rows)}};
setting the value to 1 would effectively force that value into the group by
operators. Is that what you meant?
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[ https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786342#comment-16786342 ] Gopal V commented on HIVE-21399: [~jcamachorodriguez]: also turn off this optimization if someone does want to force this via "hive.map.aggr.hash.min.reduction=1.0f;" > Adjust hive.map.aggr.hash.min.reduction statically depending on group by > statistics > --- > > Key: HIVE-21399 > URL: https://issues.apache.org/jira/browse/HIVE-21399 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21399.01.patch, HIVE-21399.02.patch > > > Currently, the value is set statically from config variable. If stats are > available, we could try to adjust this value at optimization time to favor > turning off hash aggregation earlier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786285#comment-16786285
]
Jesus Camacho Rodriguez commented on HIVE-21399:
[~vgarg], yes, that is the check on {{Mode.HASH}}.
[~gopalv], sure, will do.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786282#comment-16786282
]
Gopal V commented on HIVE-21399:
Testing the patch - the feature is right now invisible.
Can you wrap
{code}
public float getMinReductionHashAggr
{code}
for explain formatted/extended?
I realize that means a lot of qfile updates, but if this is planning wrong, the
explain should catch it.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21399) Adjust hive.map.aggr.hash.min.reduction statically depending on group by statistics
[
https://issues.apache.org/jira/browse/HIVE-21399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786280#comment-16786280
]
Vineet Garg commented on HIVE-21399:
[~jcamachorodriguez] Shouldn't {{SetHashGroupByMinReduction}} be only run for
map side aggregation? May be I missed something but I don't see the check.
> Adjust hive.map.aggr.hash.min.reduction statically depending on group by
> statistics
> ---
>
> Key: HIVE-21399
> URL: https://issues.apache.org/jira/browse/HIVE-21399
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-21399.01.patch
>
>
> Currently, the value is set statically from config variable. If stats are
> available, we could try to adjust this value at optimization time to favor
> turning off hash aggregation earlier.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
