[
https://issues.apache.org/jira/browse/TEZ-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14557976#comment-14557976
]
Jeff Zhang commented on TEZ-2475:
---------------------------------
[~fs111] It looks like TezChild hangs there for getting task from AM.
{noformat}
"LocalTaskExecutionThread [0]" daemon prio=5 tid=0x00007fe36db49000
nid=0x221170b waiting on condition [0x00000001186f5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d3f068d0> (a
com.google.common.util.concurrent.ListenableFutureTask)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
at java.util.concurrent.FutureTask.get(FutureTask.java:187)
at org.apache.tez.runtime.task.TezChild.run(TezChild.java:189)
at
org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.call(LocalContainerLauncher.java:318)
at
org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.call(LocalContainerLauncher.java:307)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
I doubt TezChild hang in this infinite loop. BTW Did you turn off some logs ?
It's weird that I didn't find any logs about the ContainerReport. If so, could
you turn on these logs and run it again ? This loop would log message so that
we can confirm whether it hangs here.
{code}
for (int idle = 1; containerTask == null; idle++) {
long sleepTimeMilliSecs = Math.min(idle * 10, getTaskMaxSleepTime);
maybeLogSleepMessage(sleepTimeMilliSecs);
TimeUnit.MILLISECONDS.sleep(sleepTimeMilliSecs);
containerTask = umbilical.getTask(containerContext);
}
{code}
> Tez local mode hanging in big testsuite
> ---------------------------------------
>
> Key: TEZ-2475
> URL: https://issues.apache.org/jira/browse/TEZ-2475
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.7.0, 0.6.1
> Reporter: André Kelpe
> Attachments: 2015-05-21_15-55-20_buildLog.log.gz
>
>
> we have a big test suite for lingual, our SQL layer for cascading. We are
> trying very hard to make it work correctly on Tez, but I am stuck:
> The setup is a huge suite of SQL based tests (6000+), which are being
> executed in order in local mode. At certain moments the whole process just
> stops. Nothing gets executed any longer. This is not all the time, but quite
> often. Note that it is not happening at the same line of code, more at
> random, which makes it quite complex to debug.
> What I am seeing, is these kind of stacktraces in the middle of the run:
> 2015-05-21 16:07:42,413 ERROR [TaskHeartbeatThread] task.TezTaskRunner
> (TezTaskRunner.java:reportError(333)) - TaskReporter reported error
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2188)
> at
> org.apache.tez.runtime.task.TaskReporter$HeartbeatCallable.call(TaskReporter.java:187)
> at
> org.apache.tez.runtime.task.TaskReporter$HeartbeatCallable.call(TaskReporter.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> This looks like it could be related to the hang, but the hang is not
> happening immediately afterwards, but some time later.
> I have gone through quite a few JIRAs and saw that there were problems with
> locks and hanging threads before, which should be fixed, but it still happens.
> I have tried 0.6.1 and 0.7.0. Both show the same behaviour.
> This gist contains a thread dump of a hanging build:
> https://gist.github.com/fs111/1ee44469bf5cc31e5a52
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
