[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[
https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632829#comment-16632829
]
Hive QA commented on HIVE-14129:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817051/HIVE-14129.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14116/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14116/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14116/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-09-29 06:14:44.661
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14116/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-09-29 06:14:44.665
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
6e27a53..0417cef master -> origin/master
+ git reset --hard HEAD
HEAD is now at 6e27a53 HIVE-20627: Concurrent async queries intermittently
fails with LockException and cause memory leak (Sankar Hariappan, reviewed by
Daniel Dai)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 0417cef HIVE-20629: Hive incremental replication fails with
events missing error if database is kept idle for more than an hour (Mahesh
Kumar Behera, reviewed by Sankar Hariappan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-09-29 06:14:46.645
+ rm -rf ../yetus_PreCommit-HIVE-Build-14116
+ mkdir ../yetus_PreCommit-HIVE-Build-14116
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14116
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14116/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/Driver.java: does not exist in
index
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/Driver.java:49
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/Driver.java' with
conflicts.
Going to apply patch with: git apply -p1
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/Driver.java:49
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/Driver.java' with
conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/Driver.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-14116
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817051 - PreCommit-HIVE-Build
> Execute move tasks in parallel
> --
>
> Key: HIVE-14129
> URL: https://issues.apache.org/jira/browse/HIVE-14129
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631689#comment-16631689 ] t oo commented on HIVE-14129: - [~ashutoshc] can this be merged to master? > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Major > Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[
https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150064#comment-16150064
]
Hive QA commented on HIVE-14129:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817051/HIVE-14129.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11022 tests
executed
*Failed tests:*
{noformat}
TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out)
(batchId=280)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=61)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=169)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6631/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6631/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6631/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817051 - PreCommit-HIVE-Build
> Execute move tasks in parallel
> --
>
> Key: HIVE-14129
> URL: https://issues.apache.org/jira/browse/HIVE-14129
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408426#comment-15408426 ] Eugene Koifman commented on HIVE-14129: --- [~thejas], which patch should I look at - here we just toggle 1 line > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408166#comment-15408166 ] Thejas M Nair commented on HIVE-14129: -- [~ashutoshc] It doesn't look like the 2nd issue is fixed. ie - bq. 2. SessionState is examined by acid code in MoveTask, if it is in a different thread that thread will not have the SessionState object available. TaskRunner would need an additional method - setSessionState, similar to setOperationLog, that sets the thread local sessionstate. Also, we should check if the methods being used in MoveTask on SessionState for acid purposes are thread safe. Looking at the method names, I would expect it to be threadsafe. cc [~ekoifman] Without that we might have issues with ACID and execparallel=true. > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[
https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369993#comment-15369993
]
Hive QA commented on HIVE-14129:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12817051/HIVE-14129.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10297 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/458/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/458/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-458/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12817051 - PreCommit-HIVE-MASTER-Build
> Execute move tasks in parallel
> --
>
> Key: HIVE-14129
> URL: https://issues.apache.org/jira/browse/HIVE-14129
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-14129.2.patch, HIVE-14129.patch, HIVE-14129.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[
https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369432#comment-15369432
]
Hive QA commented on HIVE-14129:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12816995/HIVE-14129.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/451/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/451/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-451/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-451/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 82b84ac HIVE-14173: NPE was thrown after enabling directsql in
the middle of session (Chaoyu Tang, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 82b84ac HIVE-14173: NPE was thrown after enabling directsql in
the middle of session (Chaoyu Tang, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12816995 - PreCommit-HIVE-MASTER-Build
> Execute move tasks in parallel
> --
>
> Key: HIVE-14129
> URL: https://issues.apache.org/jira/browse/HIVE-14129
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-14129.patch, HIVE-14129.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368758#comment-15368758 ] Ashutosh Chauhan commented on HIVE-14129: - Motivation is executing movetask in parallel is huge perf gain when executing on FS which are slow. > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan > Attachments: HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14129) Execute move tasks in parallel
[ https://issues.apache.org/jira/browse/HIVE-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368756#comment-15368756 ] Ashutosh Chauhan commented on HIVE-14129: - [~thejas] You pointed out couple of issues on HIVE-9665 Can you comment if they are resolved and thus enabling this will be safe now or those are still unresolved. > Execute move tasks in parallel > -- > > Key: HIVE-14129 > URL: https://issues.apache.org/jira/browse/HIVE-14129 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Reporter: Ashutosh Chauhan > Attachments: HIVE-14129.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
