[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[ https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333224#comment-16333224 ] Laszlo Bodor commented on HIVE-18445: - It seems like auto_join25 test passed, and it has also passed locally without error. [~kgyrtkirk] > qtests: auto_join25.q fails permanently > --- > > Key: HIVE-18445 > URL: https://issues.apache.org/jira/browse/HIVE-18445 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-18445.01.patch > > > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] > (batchId=72) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[
https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333068#comment-16333068
]
Hive QA commented on HIVE-18445:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12906820/HIVE-18445.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 11628 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.metastore.TestMetastoreExpr.testPartitionExpr
(batchId=257)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hive.hcatalog.common.TestHiveClientCache.testCacheExpiry
(batchId=198)
org.apache.hive.hcatalog.common.TestHiveClientCache.testCacheHit (batchId=198)
org.apache.hive.hcatalog.common.TestHiveClientCache.testCacheMiss (batchId=198)
org.apache.hive.hcatalog.common.TestHiveClientCache.testCloseAllClients
(batchId=198)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8710/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8710/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8710/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12906820 - PreCommit-HIVE-Build
> qtests: auto_join25.q fails permanently
> ---
>
> Key: HIVE-18445
> URL: https://issues.apache.org/jira/browse/HIVE-18445
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18445.01.patch
>
>
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25]
> (batchId=72)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[
https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333015#comment-16333015
]
Hive QA commented on HIVE-18445:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 6d890fa |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8710/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> qtests: auto_join25.q fails permanently
> ---
>
> Key: HIVE-18445
> URL: https://issues.apache.org/jira/browse/HIVE-18445
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18445.01.patch
>
>
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25]
> (batchId=72)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[ https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332309#comment-16332309 ] Zoltan Haindrich commented on HIVE-18445: - oh...I was looking thru things for a few minutes...but eventually I've figured it outthe groupby is added because HIVE-13567 have enable the colstats collection which technically an extra aggregator executed on the newly created table's contents - and because of the aggregation there is a groupby; because UDAFs has a wider range of function than just aggregate a column; probably most importantly, the [aggregation modes|https://github.com/apache/hive/blob/d6c6d96f98d998298618a643bd0d571d4e290dc3/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java#L76]. +1 pending tests > qtests: auto_join25.q fails permanently > --- > > Key: HIVE-18445 > URL: https://issues.apache.org/jira/browse/HIVE-18445 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-18445.01.patch > > > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] > (batchId=72) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[ https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332240#comment-16332240 ] Laszlo Bodor commented on HIVE-18445: - Hi [~kgyrtkirk] ! Thx, it was really helpful. I'll upload a patch, which seems to fix that in the way you've explained (consistently fails for all 3 queries in case of using hive.mapjoin.followby.gby.localtask.max.memory.usage). However, according to the documentation, the "group by" part is not clear to me. ["when this map join is followed by a group by"|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.mapjoin.followby.gby.localtask.max.memory.usage] Is there an implicit "group by", somewhere? I can't see any grouping in auto_join25.q. > qtests: auto_join25.q fails permanently > --- > > Key: HIVE-18445 > URL: https://issues.apache.org/jira/browse/HIVE-18445 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] > (batchId=72) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[
https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326105#comment-16326105
]
Zoltan Haindrich commented on HIVE-18445:
-
I was looking into whether this feature works for producing test cases for
reoptimization...and I've probably just bumped into the reason this test have
changed after HIVE-13567:
[https://github.com/apache/hive/blob/6938fcabf0944817fed8241c48b57fb3f5d98c69/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java#L133
which is a feature introduced in HIVE-1830
so probably setting {{hive.mapjoin.followby.gby.localtask.max.memory.usage}} to
some low value would probably re-introduce the backuptasks
> qtests: auto_join25.q fails permanently
> ---
>
> Key: HIVE-18445
> URL: https://issues.apache.org/jira/browse/HIVE-18445
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25]
> (batchId=72)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[ https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326030#comment-16326030 ] Zoltan Haindrich commented on HIVE-18445: - query plans look good to me ; I've see no changes in the select join type (other changes are expect after HIVE-13567) ; interestingly: adding these explains to the q file made the "Execution Error" messages go away somehow > qtests: auto_join25.q fails permanently > --- > > Key: HIVE-18445 > URL: https://issues.apache.org/jira/browse/HIVE-18445 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > > org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] > (batchId=72) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[
https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326021#comment-16326021
]
Zoltan Haindrich commented on HIVE-18445:
-
it seems like the test was added in HIVE-1642; with the intention to check for
the "backuptask" doing the job in case the first one fails...I think the test
should have ~3 BackupTask executionions in the q.out; but instead there is 1
In the light of the above; I've looked into when these {{BackupTask}} attempts
are changed - and it looks like after HIVE-13567 some of the "Execution
Error"-s are gone; probably the plans have changed for these queries
> qtests: auto_join25.q fails permanently
> ---
>
> Key: HIVE-18445
> URL: https://issues.apache.org/jira/browse/HIVE-18445
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25]
> (batchId=72)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[
https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325504#comment-16325504
]
Laszlo Bodor commented on HIVE-18445:
-
Or we can completely turn of auto join for statements out of scope, the test
case will test conditonal task and exhaustion anyway. The current problem is,
sometimes there are repeated errors messages in q.out. Further investigation is
in progress.
q file
{code}
set hive.auto.convert.join = true;
FROM src src1 JOIN src src2 ON (src1.key = src2.key) JOIN src src3 ON (src1.key
+ src2.key = src3.key)
INSERT OVERWRITE TABLE dest_j2 SELECT src1.key, src3.value;
set hive.auto.convert.join = false;
{code}
q.out file
{code}
PREHOOK: query: FROM src src1 JOIN src src2 ON (src1.key = src2.key) JOIN src
src3 ON (src1.key + src2.key = src3.key)
INSERT OVERWRITE TABLE dest_j2 SELECT src1.key, src3.value
PREHOOK: type: QUERY
PREHOOK: Input: default@src
PREHOOK: Output: default@dest_j2
*Hive Runtime Error: Map local work exhausted memory
FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask
Hive Runtime Error: Map local work exhausted memory
FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask*
POSTHOOK: query: FROM src src1 JOIN src src2 ON (src1.key = src2.key) JOIN src
src3 ON (src1.key + src2.key = src3.key)
INSERT OVERWRITE TABLE dest_j2 SELECT src1.key, src3.value
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src
POSTHOOK: Output: default@dest_j2
{code}
> qtests: auto_join25.q fails permanently
> ---
>
> Key: HIVE-18445
> URL: https://issues.apache.org/jira/browse/HIVE-18445
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25]
> (batchId=72)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[
https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325380#comment-16325380
]
Laszlo Bodor commented on HIVE-18445:
-
The issue here is, that the test is about exhausting the local mapper task's
memory, and to achieve this, it sets a parameter at the beginning...
{code}
set hive.mapjoin.localtask.max.memory.usage = 0.0001;
{code}
...so the task can use the 0.0001% percent of the process' memory. It seems to
be ok for testing memory exhaustion, but the problem is that it affects all
queries.
Checking the q.out file, it seems like we expect an exhaustion by running the
2nd query:
{code}
FROM src src1 JOIN src src2 ON (src1.key = src2.key) JOIN src src3 ON (src1.key
+ src2.key = src3.key)
INSERT OVERWRITE TABLE dest_j2 SELECT src1.key, src3.value;
{code}
But when the test fails, it fails on the first statement (which is not supposed
to fail):
{code}
FROM srcpart src1 JOIN src src2 ON (src1.key = src2.key)
INSERT OVERWRITE TABLE dest1 SELECT src1.key, src2.value
where (src1.ds = '2008-04-08' or src1.ds = '2008-04-09' )and (src1.hr = '12' or
src1.hr = '11');
{code}
I think the best practise would be to set the parameter before the target
query, and reset it to default (or a higher value) after, like:
{code}
set hive.mapjoin.localtask.max.memory.usage = 0.0001;
FROM src src1 JOIN src src2 ON (src1.key = src2.key) JOIN src src3 ON (src1.key
+ src2.key = src3.key)
INSERT OVERWRITE TABLE dest_j2 SELECT src1.key, src3.value;
set hive.mapjoin.localtask.max.memory.usage = 0.9;
{code}
> qtests: auto_join25.q fails permanently
> ---
>
> Key: HIVE-18445
> URL: https://issues.apache.org/jira/browse/HIVE-18445
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25]
> (batchId=72)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18445) qtests: auto_join25.q fails permanently
[
https://issues.apache.org/jira/browse/HIVE-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325351#comment-16325351
]
Laszlo Bodor commented on HIVE-18445:
-
It seems like the results don't depend on other qtests, as it's sometimes fails
when running separately.
{code}
mvn test -Dtest=TestCliDriver -Pitests -e -pl itests/qtest -Dqfile=auto_join25.q
{code}
In the event of failure, the q.out contains something like:
{code}
< Hive Runtime Error: Map local work exhausted memory
< FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
< ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask
{code}
> qtests: auto_join25.q fails permanently
> ---
>
> Key: HIVE-18445
> URL: https://issues.apache.org/jira/browse/HIVE-18445
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25]
> (batchId=72)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
