[
https://issues.apache.org/jira/browse/MAPREDUCE-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925186#action_12925186
]
Joydeep Sen Sarma commented on MAPREDUCE-2157:
----------------------------------------------
i think i have a idea now on why InterruptedExceptions can also happen
unexpectedly. Consider this hadoop-20 TT code:
{quote}
try {
...
while (numFreeSlots.get() < task.getNumSlotsRequired()) {
LOG.info("...");
numFreeSlots.wait();
}
catch (InterruptedException e) {
return;
}
{quote}
LOG.info can set the thread interrupted state because it can get an
InterruptedIOException. InterruptedIOException can be thrown for reasons other
than threads getting interrupted via Java Thread.interrupt() invocations.
Documentation seems to suggest that it will be thrown on interrupted read/write
system calls (which log4j does plenty of).
If thread interrupted state is set - then the succeeding numFreeSlots.wait()
call will throw an InterruptedException. This is as per documentation. Bingo -
thread will now terminate.
I think the bottomline is that we should *never* depend solely on
InterruptedException of interrupted status to determine shutdown. it might be
ok if the only thing that the code does is Object.wait() and has while/try
blocks around it - but if file/socket operations are involved in the mix - this
is very dangerous.
> tasklauncher threads in TaskTracker can die because of unexpected interrupts
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-2157
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2157
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Priority: Critical
>
> taskLauncher thread exits on interruptedException and on Interrupt conditions
> without checking for any shutdown flag:
> while (!Thread.interrupted()) {
> ...
> } catch (InterruptedException e) {
> return; // ALL DONE
>
> }
> }
> If the interrupt happened because of reasons other than TaskTracker.close() -
> then the TaskTracker will look functional - but will not be able to schedule
> tasks anymore. worse - some tasks (that are in the launch queue) will hang
> indefinitely un UNASSIGNED state (the JobTracker will not even time them
> out). We have seen this cause jobs to hang indefinitely.
> It seems that the interrupted condition can be set by log4j (of which there
> are many calls inside TaskLauncher). See or instance:
> http://logging.apache.org/log4j/1.2/xref/org/apache/log4j/AsyncAppender.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
