[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[ https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887057#comment-15887057 ] Siddharth Seth commented on HIVE-15958: --- +1 > LLAP: IPC connections are not being reused for umbilical protocol > - > > Key: HIVE-15958 > URL: https://issues.apache.org/jira/browse/HIVE-15958 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15958.1.patch, HIVE-15958.2.patch, > HIVE-15958.3.patch, HIVE-15958.4.patch, HIVE-15958.4.patch, HIVE-15958.5.patch > > > During concurrency testing, observed 1000s of ipc thread creations. Ideally, > the connections to same hosts should be reused. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[
https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884263#comment-15884263
]
Hive QA commented on HIVE-15958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854633/HIVE-15958.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10259 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=211)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3782/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3782/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3782/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12854633 - PreCommit-HIVE-Build
> LLAP: IPC connections are not being reused for umbilical protocol
> -
>
> Key: HIVE-15958
> URL: https://issues.apache.org/jira/browse/HIVE-15958
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15958.1.patch, HIVE-15958.2.patch,
> HIVE-15958.3.patch, HIVE-15958.4.patch, HIVE-15958.4.patch, HIVE-15958.5.patch
>
>
> During concurrency testing, observed 1000s of ipc thread creations. Ideally,
> the connections to same hosts should be reused.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[
https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884125#comment-15884125
]
Hive QA commented on HIVE-15958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854633/HIVE-15958.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10259 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_bitmap_auto_partitioned]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=140)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3773/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3773/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3773/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12854633 - PreCommit-HIVE-Build
> LLAP: IPC connections are not being reused for umbilical protocol
> -
>
> Key: HIVE-15958
> URL: https://issues.apache.org/jira/browse/HIVE-15958
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15958.1.patch, HIVE-15958.2.patch,
> HIVE-15958.3.patch, HIVE-15958.4.patch, HIVE-15958.4.patch, HIVE-15958.5.patch
>
>
> During concurrency testing, observed 1000s of ipc thread creations. Ideally,
> the connections to same hosts should be reused.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[ https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883647#comment-15883647 ] Siddharth Seth commented on HIVE-15958: --- The stopUmbilical in hasAmFaield() - is this required? Think this has the potential to cause an NPE if there's a taskKilled already queued up. In QueryTracker - instead of introducing a new map to store the NodeId - the NodeId can be stored in QueryInfo. Otherwise, looks good. > LLAP: IPC connections are not being reused for umbilical protocol > - > > Key: HIVE-15958 > URL: https://issues.apache.org/jira/browse/HIVE-15958 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15958.1.patch, HIVE-15958.2.patch, > HIVE-15958.3.patch, HIVE-15958.4.patch, HIVE-15958.4.patch > > > During concurrency testing, observed 1000s of ipc thread creations. Ideally, > the connections to same hosts should be reused. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[
https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882470#comment-15882470
]
Hive QA commented on HIVE-15958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854319/HIVE-15958.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3750/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3750/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3750/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-02-24 11:14:06.379
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3750/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-02-24 11:14:06.381
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 338a7ee HIVE-15993 : Hive REPL STATUS is not returning last
event ID (Sankar Hariappan, reviewed by Sushanth Sowmyan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 338a7ee HIVE-15993 : Hive REPL STATUS is not returning last
event ID (Sankar Hariappan, reviewed by Sushanth Sowmyan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-02-24 11:14:07.396
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java:
No such file or directory
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
No such file or directory
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java:
No such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12854319 - PreCommit-HIVE-Build
> LLAP: IPC connections are not being reused for umbilical protocol
> -
>
> Key: HIVE-15958
> URL: https://issues.apache.org/jira/browse/HIVE-15958
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15958.1.patch, HIVE-15958.2.patch,
> HIVE-15958.3.patch, HIVE-15958.4.patch
>
>
> During concurrency testing, observed 1000s of ipc thread creations. Ideally,
> the connections to same hosts should be reused.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[
https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881998#comment-15881998
]
Hive QA commented on HIVE-15958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854319/HIVE-15958.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3742/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3742/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3742/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-02-24 05:30:55.487
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3742/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-02-24 05:30:55.490
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
4f18acd..338a7ee master -> origin/master
+ git reset --hard HEAD
HEAD is now at 4f18acd HIVE-15668 : change REPL DUMP syntax to use "LIMIT"
instead of "BATCH" keyword
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 338a7ee HIVE-15993 : Hive REPL STATUS is not returning last
event ID (Sankar Hariappan, reviewed by Sushanth Sowmyan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-02-24 05:30:58.390
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java:
No such file or directory
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
No such file or directory
error:
a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java:
No such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12854319 - PreCommit-HIVE-Build
> LLAP: IPC connections are not being reused for umbilical protocol
> -
>
> Key: HIVE-15958
> URL: https://issues.apache.org/jira/browse/HIVE-15958
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15958.1.patch, HIVE-15958.2.patch,
> HIVE-15958.3.patch, HIVE-15958.4.patch
>
>
> During concurrency testing, observed 1000s of ipc thread creations. Ideally,
> the connections to same hosts should be reused.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[ https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881180#comment-15881180 ] Prasanth Jayachandran commented on HIVE-15958: -- Just noticed another race between cleanup and kill task. Will fix it shortly. > LLAP: IPC connections are not being reused for umbilical protocol > - > > Key: HIVE-15958 > URL: https://issues.apache.org/jira/browse/HIVE-15958 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15958.1.patch, HIVE-15958.2.patch, > HIVE-15958.3.patch > > > During concurrency testing, observed 1000s of ipc thread creations. Ideally, > the connections to same hosts should be reused. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[ https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879895#comment-15879895 ] Siddharth Seth commented on HIVE-15958: --- I suspect the following is the reason for connections not being re-used for taskKilled. For regular heartbeats, only one session will ever run for an AM - and this is controlled via the QueueCallable / HeartbeatCallable. When taskKilled comes into play, it is possible for a taskKilled to get a handle on the umbilical, and have one of the queued threads close the umbilical right after that, resulting in an error. We have that situation again. More prominent now - since queryComplete causes fragments to be killed (should probably not be done - HIVE-16021), which in turn result in a heartbeat. The queryComplete closes the umbilical, while taskKilled requests get scheduled. Also, iterating over the knownAppMasters is very avoidable. We can store information about the AM in the queryTracker, and retrieve it on queryComplete. Alternately send the AM information on the queryComplete call. > LLAP: IPC connections are not being reused for umbilical protocol > - > > Key: HIVE-15958 > URL: https://issues.apache.org/jira/browse/HIVE-15958 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15958.1.patch, HIVE-15958.2.patch > > > During concurrency testing, observed 1000s of ipc thread creations. Ideally, > the connections to same hosts should be reused. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[
https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877758#comment-15877758
]
Hive QA commented on HIVE-15958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12853868/HIVE-15958.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10252 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3687/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3687/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3687/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12853868 - PreCommit-HIVE-Build
> LLAP: IPC connections are not being reused for umbilical protocol
> -
>
> Key: HIVE-15958
> URL: https://issues.apache.org/jira/browse/HIVE-15958
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15958.1.patch, HIVE-15958.2.patch
>
>
> During concurrency testing, observed 1000s of ipc thread creations. Ideally,
> the connections to same hosts should be reused.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[
https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876638#comment-15876638
]
Hive QA commented on HIVE-15958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12853191/HIVE-15958.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10251 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
(batchId=152)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31]
(batchId=133)
org.apache.hadoop.hive.metastore.TestRemoteUGIHiveMetaStoreIpAddress.testIpAddress
(batchId=201)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=211)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3672/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3672/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3672/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12853191 - PreCommit-HIVE-Build
> LLAP: IPC connections are not being reused for umbilical protocol
> -
>
> Key: HIVE-15958
> URL: https://issues.apache.org/jira/browse/HIVE-15958
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15958.1.patch
>
>
> During concurrency testing, observed 1000s of ipc thread creations. Ideally,
> the connections to same hosts should be reused.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[
https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871406#comment-15871406
]
Hive QA commented on HIVE-15958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12853191/HIVE-15958.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 10244 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
(batchId=152)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_external_acid]
(batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_not_acid]
(batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_view_failure1]
(batchId=85)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_view_failure2]
(batchId=85)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[external1]
(batchId=85)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_count_distinct]
(batchId=106)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterTable
(batchId=194)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=194)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testAlterTable
(batchId=197)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testTransactionalValidation
(batchId=197)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterTable
(batchId=193)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTransactionalValidation
(batchId=193)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testAlterTable
(batchId=191)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testTransactionalValidation
(batchId=191)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testAlterTable
(batchId=202)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testTransactionalValidation
(batchId=202)
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
(batchId=210)
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
(batchId=208)
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
(batchId=220)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=211)
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands (batchId=170)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=170)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createTable
(batchId=221)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
(batchId=213)
org.apache.hive.service.cli.thrift.TestThriftCLIServiceWithBinary.testExecuteStatementAsync
(batchId=213)
org.apache.hive.service.cli.thrift.TestThriftCLIServiceWithHttp.testExecuteStatementAsync
(batchId=213)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3618/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3618/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3618/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 33 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12853191 - PreCommit-HIVE-Build
> LLAP: IPC connections are not being reused for umbilical protocol
> -
>
> Key: HIVE-15958
> URL: https://issues.apache.org/jira/browse/HIVE-15958
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-15958.1.patch
>
>
> During concurrency testing, observed 1000s of ipc thread creations. Ideally,
> the connections to same hosts should be reused.
--
This message was sent by
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[ https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871129#comment-15871129 ] Prasanth Jayachandran commented on HIVE-15958: -- [~sseth] created HIVE-15960 for doing the cleanup in query completion. > LLAP: IPC connections are not being reused for umbilical protocol > - > > Key: HIVE-15958 > URL: https://issues.apache.org/jira/browse/HIVE-15958 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15958.1.patch > > > During concurrency testing, observed 1000s of ipc thread creations. Ideally, > the connections to same hosts should be reused. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[ https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871116#comment-15871116 ] Siddharth Seth commented on HIVE-15958: --- Looks good. There's still a chance of a bunch of extra connections being made if AMNodeInfo is not cached. It gets removed when fragmentCount goes to 0, i.e. all fragments for this AM are complete or executing (not in the wait queue). Would be good to clean the AMNodeInfo cache on query complete instead of count going to 0. Can be a follow up jira though, since this fixes the main issue. [~gopalv] - this does not undo that change. This is related to unnecessary connection being created for kill messages. Tasks still use their own threads for heartbeats. Wasn't HADOOP-11772 related to not being able to get a connection, rather than everything serialized on a single connection? I think we still need to move to everything on a single connection at some point. > LLAP: IPC connections are not being reused for umbilical protocol > - > > Key: HIVE-15958 > URL: https://issues.apache.org/jira/browse/HIVE-15958 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15958.1.patch > > > During concurrency testing, observed 1000s of ipc thread creations. Ideally, > the connections to same hosts should be reused. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15958) LLAP: IPC connections are not being reused for umbilical protocol
[ https://issues.apache.org/jira/browse/HIVE-15958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871099#comment-15871099 ] Gopal V commented on HIVE-15958: [~prasanth_j]: does this undo the fix in HADOOP-11772? We had the exact opposite problem of synchronizing behind a single Client for all heartbeats & events. > LLAP: IPC connections are not being reused for umbilical protocol > - > > Key: HIVE-15958 > URL: https://issues.apache.org/jira/browse/HIVE-15958 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Rajesh Balamohan >Assignee: Prasanth Jayachandran > Attachments: HIVE-15958.1.patch > > > During concurrency testing, observed 1000s of ipc thread creations. Ideally, > the connections to same hosts should be reused. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
