[jira] [Commented] (HIVE-15889) LLAP: Some tasks still run after hive cli is shutdown
[
https://issues.apache.org/jira/browse/HIVE-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865108#comment-15865108
]
Thejas M Nair commented on HIVE-15889:
--
Why are we having to use reflection here ?
(I could be missing something that should be obvious, looks like was introduced
long ago in HIVE-8393)
> LLAP: Some tasks still run after hive cli is shutdown
> -
>
> Key: HIVE-15889
> URL: https://issues.apache.org/jira/browse/HIVE-15889
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Fix For: 2.2.0
>
> Attachments: HIVE-15889.1.patch, HIVE-15889.2.patch,
> HIVE-15889.3.patch
>
>
> E.g: In cross product case, the tight loop in merge join operator ignores any
> interrupt or abort flag checks, causing the tasks to be in running state even
> when the client cli is quit.
> intensionally written cross product query to simulate this.
> {noformat}
> hive> select count(1) from lineitem, orders;;
> {noformat}
> Even when the cli is quit, LLAP would continue executing the task for quite
> sometime. E.g stack trace
> {noformat}
> "TezTaskRunner" #1945 daemon prio=5 os_prio=0 tid=0x7fe9e43a5000
> nid=0x4c8 runnable [0x7fc8d881b000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:453)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.mergeJoinComputeKeys(CommonMergeJoinOperator.java:603)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:207)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:410)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:381)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:209)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15889) LLAP: Some tasks still run after hive cli is shutdown
[
https://issues.apache.org/jira/browse/HIVE-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864924#comment-15864924
]
Sergey Shelukhin commented on HIVE-15889:
-
+1
> LLAP: Some tasks still run after hive cli is shutdown
> -
>
> Key: HIVE-15889
> URL: https://issues.apache.org/jira/browse/HIVE-15889
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-15889.1.patch, HIVE-15889.2.patch,
> HIVE-15889.3.patch
>
>
> E.g: In cross product case, the tight loop in merge join operator ignores any
> interrupt or abort flag checks, causing the tasks to be in running state even
> when the client cli is quit.
> intensionally written cross product query to simulate this.
> {noformat}
> hive> select count(1) from lineitem, orders;;
> {noformat}
> Even when the cli is quit, LLAP would continue executing the task for quite
> sometime. E.g stack trace
> {noformat}
> "TezTaskRunner" #1945 daemon prio=5 os_prio=0 tid=0x7fe9e43a5000
> nid=0x4c8 runnable [0x7fc8d881b000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:453)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.mergeJoinComputeKeys(CommonMergeJoinOperator.java:603)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:207)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:410)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:381)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:209)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15889) LLAP: Some tasks still run after hive cli is shutdown
[
https://issues.apache.org/jira/browse/HIVE-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864875#comment-15864875
]
Hive QA commented on HIVE-15889:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852443/HIVE-15889.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10238 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=140)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3525/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3525/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3525/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12852443 - PreCommit-HIVE-Build
> LLAP: Some tasks still run after hive cli is shutdown
> -
>
> Key: HIVE-15889
> URL: https://issues.apache.org/jira/browse/HIVE-15889
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
> Attachments: HIVE-15889.1.patch, HIVE-15889.2.patch,
> HIVE-15889.3.patch
>
>
> E.g: In cross product case, the tight loop in merge join operator ignores any
> interrupt or abort flag checks, causing the tasks to be in running state even
> when the client cli is quit.
> intensionally written cross product query to simulate this.
> {noformat}
> hive> select count(1) from lineitem, orders;;
> {noformat}
> Even when the cli is quit, LLAP would continue executing the task for quite
> sometime. E.g stack trace
> {noformat}
> "TezTaskRunner" #1945 daemon prio=5 os_prio=0 tid=0x7fe9e43a5000
> nid=0x4c8 runnable [0x7fc8d881b000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:453)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.mergeJoinComputeKeys(CommonMergeJoinOperator.java:603)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:207)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:410)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:381)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:209)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15889) LLAP: Some tasks still run after hive cli is shutdown
[
https://issues.apache.org/jira/browse/HIVE-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864345#comment-15864345
]
Sergey Shelukhin commented on HIVE-15889:
-
Should TezJobExecHelper use setAccessible to avoid being affected by this? Also
why does it have to use reflection? Also should it cache the method
statically.. :)
> LLAP: Some tasks still run after hive cli is shutdown
> -
>
> Key: HIVE-15889
> URL: https://issues.apache.org/jira/browse/HIVE-15889
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: HIVE-15889.1.patch, HIVE-15889.2.patch
>
>
> E.g: In cross product case, the tight loop in merge join operator ignores any
> interrupt or abort flag checks, causing the tasks to be in running state even
> when the client cli is quit.
> intensionally written cross product query to simulate this.
> {noformat}
> hive> select count(1) from lineitem, orders;;
> {noformat}
> Even when the cli is quit, LLAP would continue executing the task for quite
> sometime. E.g stack trace
> {noformat}
> "TezTaskRunner" #1945 daemon prio=5 os_prio=0 tid=0x7fe9e43a5000
> nid=0x4c8 runnable [0x7fc8d881b000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:453)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.mergeJoinComputeKeys(CommonMergeJoinOperator.java:603)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:207)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:410)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:381)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:209)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15889) LLAP: Some tasks still run after hive cli is shutdown
[
https://issues.apache.org/jira/browse/HIVE-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863304#comment-15863304
]
Hive QA commented on HIVE-15889:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852299/HIVE-15889.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10237 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3514/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3514/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3514/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12852299 - PreCommit-HIVE-Build
> LLAP: Some tasks still run after hive cli is shutdown
> -
>
> Key: HIVE-15889
> URL: https://issues.apache.org/jira/browse/HIVE-15889
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: HIVE-15889.1.patch
>
>
> E.g: In cross product case, the tight loop in merge join operator ignores any
> interrupt or abort flag checks, causing the tasks to be in running state even
> when the client cli is quit.
> intensionally written cross product query to simulate this.
> {noformat}
> hive> select count(1) from lineitem, orders;;
> {noformat}
> Even when the cli is quit, LLAP would continue executing the task for quite
> sometime. E.g stack trace
> {noformat}
> "TezTaskRunner" #1945 daemon prio=5 os_prio=0 tid=0x7fe9e43a5000
> nid=0x4c8 runnable [0x7fc8d881b000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:453)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.mergeJoinComputeKeys(CommonMergeJoinOperator.java:603)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:207)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:410)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:381)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:209)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15889) LLAP: Some tasks still run after hive cli is shutdown
[
https://issues.apache.org/jira/browse/HIVE-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863436#comment-15863436
]
Hive QA commented on HIVE-15889:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852318/HIVE-15889.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10237 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3516/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3516/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3516/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12852318 - PreCommit-HIVE-Build
> LLAP: Some tasks still run after hive cli is shutdown
> -
>
> Key: HIVE-15889
> URL: https://issues.apache.org/jira/browse/HIVE-15889
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: HIVE-15889.1.patch, HIVE-15889.2.patch
>
>
> E.g: In cross product case, the tight loop in merge join operator ignores any
> interrupt or abort flag checks, causing the tasks to be in running state even
> when the client cli is quit.
> intensionally written cross product query to simulate this.
> {noformat}
> hive> select count(1) from lineitem, orders;;
> {noformat}
> Even when the cli is quit, LLAP would continue executing the task for quite
> sometime. E.g stack trace
> {noformat}
> "TezTaskRunner" #1945 daemon prio=5 os_prio=0 tid=0x7fe9e43a5000
> nid=0x4c8 runnable [0x7fc8d881b000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:453)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.mergeJoinComputeKeys(CommonMergeJoinOperator.java:603)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:207)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:410)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:381)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:209)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15889) LLAP: Some tasks still run after hive cli is shutdown
[
https://issues.apache.org/jira/browse/HIVE-15889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863255#comment-15863255
]
Rajesh Balamohan commented on HIVE-15889:
-
originally reported by [~gopalv].
> LLAP: Some tasks still run after hive cli is shutdown
> -
>
> Key: HIVE-15889
> URL: https://issues.apache.org/jira/browse/HIVE-15889
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>
> E.g: In cross product case, the tight loop in merge join operator ignores any
> interrupt or abort flag checks, causing the tasks to be in running state even
> when the client cli is quit.
> intensionally written cross product query to simulate this.
> {noformat}
> hive> select count(1) from lineitem, orders;;
> {noformat}
> Even when the cli is quit, LLAP would continue executing the task for quite
> sometime. E.g stack trace
> {noformat}
> "TezTaskRunner" #1945 daemon prio=5 os_prio=0 tid=0x7fe9e43a5000
> nid=0x4c8 runnable [0x7fc8d881b000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:453)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.mergeJoinComputeKeys(CommonMergeJoinOperator.java:603)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:207)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchOneRow(CommonMergeJoinOperator.java:410)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.fetchNextGroup(CommonMergeJoinOperator.java:381)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.doFirstFetchIfNeeded(CommonMergeJoinOperator.java:491)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:209)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:351)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:282)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:319)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
