[
https://issues.apache.org/jira/browse/TEZ-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571582#comment-14571582
]
Siddharth Seth commented on TEZ-2475:
-------------------------------------
[~fs111] - I got two runs - one of which went to test ~5700, but I had to kill
it while it was making progress. The second one ran through fine. Both were
against 0.6.2-SNAPSHOT with the patch on this jira applied (which adds logging
only).
{code}
Generating HTML test report...
Finished generating test html results (3.678 secs) into:
/Users/sseth/Downloads/bugs/2475/comprehensive/comprehensive-hadoop2-tez/build/reports/tests
:comprehensive-hadoop2-tez:test (Thread[main,5,main]) completed. Took 4 hrs 58
mins 13.032 secs.
BUILD SUCCESSFUL
Total time: 4 hrs 58 mins 28.064 secs
{code}
Do you see any successful runs, or does it always hang ?
I did see a hang on 0.7 - which also showed some exceptions from the
PipelinedSorter (BufferTooSmall). I'll try running against 0.7 again later in
the day and see what happens.
I don't see Tez logs anywhere. Anything specific to be done to generate these ?
Also, if a dag fails - I assume the test suite would fail there - rather than
submitting additional DAGs to the same client ?
> Tez local mode hanging in big testsuite
> ---------------------------------------
>
> Key: TEZ-2475
> URL: https://issues.apache.org/jira/browse/TEZ-2475
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.7.0, 0.6.1
> Reporter: André Kelpe
> Attachments: 2015-05-21_15-55-20_buildLog.log.gz, TEZ-2475.debug.1.txt
>
>
> we have a big test suite for lingual, our SQL layer for cascading. We are
> trying very hard to make it work correctly on Tez, but I am stuck:
> The setup is a huge suite of SQL based tests (6000+), which are being
> executed in order in local mode. At certain moments the whole process just
> stops. Nothing gets executed any longer. This is not all the time, but quite
> often. Note that it is not happening at the same line of code, more at
> random, which makes it quite complex to debug.
> What I am seeing, is these kind of stacktraces in the middle of the run:
> 2015-05-21 16:07:42,413 ERROR [TaskHeartbeatThread] task.TezTaskRunner
> (TezTaskRunner.java:reportError(333)) - TaskReporter reported error
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2188)
> at
> org.apache.tez.runtime.task.TaskReporter$HeartbeatCallable.call(TaskReporter.java:187)
> at
> org.apache.tez.runtime.task.TaskReporter$HeartbeatCallable.call(TaskReporter.java:118)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> This looks like it could be related to the hang, but the hang is not
> happening immediately afterwards, but some time later.
> I have gone through quite a few JIRAs and saw that there were problems with
> locks and hanging threads before, which should be fixed, but it still happens.
> I have tried 0.6.1 and 0.7.0. Both show the same behaviour.
> This gist contains a thread dump of a hanging build:
> https://gist.github.com/fs111/1ee44469bf5cc31e5a52
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
