[jira] [Commented] (HIVE-22198) Execute unoin-all with childs Join in parallel
[
https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946582#comment-16946582
]
Hive QA commented on HIVE-22198:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12980846/HIVE-22198.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18903/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18903/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18903/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-10-08 07:30:51.472
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-18903/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-10-08 07:30:51.475
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 050f918 HIVE-22252: Fix caught NullPointerExceptions generated
during EXPLAIN (John Sherman, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 050f918 HIVE-22252: Fix caught NullPointerExceptions generated
during EXPLAIN (John Sherman, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-10-08 07:30:53.008
+ rm -rf ../yetus_PreCommit-HIVE-Build-18903
+ mkdir ../yetus_PreCommit-HIVE-Build-18903
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-18903
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-18903/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/Driver.java: does not exist in
index
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/Driver.java:2881
error: repository lacks the necessary blob to fall back on 3-way merge.
error: ql/src/java/org/apache/hadoop/hive/ql/Driver.java: patch does not apply
error: src/java/org/apache/hadoop/hive/ql/Driver.java: does not exist in index
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-18903
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12980846 - PreCommit-HIVE-Build
> Execute unoin-all with childs Join in parallel
> --
>
> Key: HIVE-22198
> URL: https://issues.apache.org/jira/browse/HIVE-22198
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: LuGuangMing
>Assignee: LuGuangMing
>Priority: Major
> Attachments: HIVE-22198.patch, image-2019-09-20-11-38-37-433.png,
> image-2019-09-20-11-39-30-347.png, image-2019-10-08-09-51-06-144.png,
> test-parallel.sql
>
>
> set parallel is true, set skewjoin is false, set auto convert join is false.
> run a unoin all, There is nothing error message, but some result data is
> missing, details check attatchment [^test-parallel.sql]
> create table tab1(tid int, com string) row format delimited fields terminated
> by '\t' stored as textfile;
> create table tab2(tid int, com string) row format delimited fields
> terminated by '\t' stored as textfile;
> create table tab3(tid int, com string) row format delimited fields
> terminated by '\t' stored as textfile;
> create table tab4(tid int, com string) row format delimited fields
> terminated by '\t' stored as textfile;
> insert into tab1
[jira] [Commented] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946392#comment-16946392 ] LuGuangMing commented on HIVE-22198: [~aditya-shah] ConditionalTask is not easily aware of the parent node that needs to be repaired, such as the following SQL scenario: create table test_parallel stored as orcfile as SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; !image-2019-10-08-09-51-06-144.png! Its scheduling order selects a branch within ConditionalTask, The parent relationship of a branch that is selected for execution can be preserved, and the parent relationship of a branch that is not selected for execution must be deleted, The branch that we want to execute is not easily perceived in advance, So I consider placing dependency repair on the parent node in the launchTask phase > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: HIVE-22198.patch, image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, image-2019-10-08-09-51-06-144.png, > test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment [^test-parallel.sql] > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935564#comment-16935564 ] Aditya Shah commented on HIVE-22198: [~luguangming], can we repair parents just in case of the conditional task instead of doing the same for all? > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: HIVE-22198.patch, image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment [^test-parallel.sql] > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934132#comment-16934132 ] LuGuangMing commented on HIVE-22198: 【repair】 Fix the parent dependencies of children stage-3 when stage-2 and stage-6 run，details check [^HIVE-22198.patch] > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: HIVE-22198.patch, image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment [^test-parallel.sql] > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22198) Execute unoin-all with childs Join in parallel
[ https://issues.apache.org/jira/browse/HIVE-22198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934016#comment-16934016 ] LuGuangMing commented on HIVE-22198: 【analyze】 explain SELECT sum(1) as a FROM tab1 t1 INNER JOIN tab2 t2 ON t1.com = t2.com UNION ALL SELECT sum(1) as a FROM tab3 t3 INNER JOIN tab4 t4 ON t3.com = t4.com; !image-2019-09-20-11-38-37-433.png|width=456,height=176!!image-2019-09-20-11-39-30-347.png|width=378,height=303! Analysis explian plan and resouce code, shows that stage-3 is added to the runnable queue when one of stage-2 or stage-6 runs is finished, due to ConditionalTask stage-9 and ConditionalTask stage-11 was invoked by parallel, remove the parent dependency of stage-3 on stage-2 and stage-6, The Driver scheduler finds that one of the stage-2 or stage-6 runtimes is finished, The children stage-3 will be found for scheduling add to runnable queue to run, At this time, there is still one node in stage-2 or stage-6 that has not run finished, and the result of the stage that has not finished will not be read by stage-3, so part of the result data is missing > Execute unoin-all with childs Join in parallel > -- > > Key: HIVE-22198 > URL: https://issues.apache.org/jira/browse/HIVE-22198 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0, 3.0.0, 3.1.0 >Reporter: LuGuangMing >Assignee: LuGuangMing >Priority: Major > Attachments: image-2019-09-20-11-38-37-433.png, > image-2019-09-20-11-39-30-347.png, test-parallel.sql > > > set parallel is true, set skewjoin is false, set auto convert join is false. > run a unoin all, There is nothing error message, but some result data is > missing, details check attatchment sql file > create table tab1(tid int, com string) row format delimited fields terminated > by '\t' stored as textfile; > create table tab2(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab3(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > create table tab4(tid int, com string) row format delimited fields > terminated by '\t' stored as textfile; > insert into tab1 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab2 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab3 values(1,'abc'),(2,'bcd'),(3,'cde'); > insert into tab4 values(1,'abc'),(2,'bcd'),(3,'cde'); > set hive.auto.convert.join=false; > set hive.optimize.skewjoin=true; > set hive.exec.parallel=true; > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > create table test_parallel stored as orcfile as > SELECT sum(1) as a > FROM tab1 t1 > INNER JOIN tab2 t2 > ON t1.com = t2.com > UNION ALL > SELECT sum(1) as a > FROM tab3 t3 > INNER JOIN tab4 t4 > ON t3.com = t4.com; > select * from test_parallel; > The result data should be two, but only one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
