[
https://issues.apache.org/jira/browse/MAPREDUCE-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494295#comment-13494295
]
Jason Lowe commented on MAPREDUCE-4751:
---------------------------------------
Part of the issue is that the job is hanging around waiting for all tasks to be
killed rather than just exiting and letting YARN shoot any straggling
containers. I think it would be simpler/safer for the AM to just write out the
final state stuff and exit, much like it does for the FAILED state. If job's
KILL_WAIT really is necessary then we'd need a corresponding FAILED_WAIT state
to handle waiting for task cleanup when a job fails.
If we don't need the job's KILL_WAIT state then similarly we can probably ditch
the task KILL_WAIT state -- it could just send off kills to all the
corresponding task attempts and sit in the KILLED state. Does it really need
to wait?
Removing KILL_WAIT is quite a bit bigger change than the current one. as it
would break a lot of tests that know and expect the KILL_WAIT state. However I
think it would be more robust in the long-term, as KILL_WAIT seems like a state
primed for hanging if we don't really need it. Since we're eager to get a fix
for this in soon we could address that in a followup JIRA.
> AM stuck in KILL_WAIT for days
> ------------------------------
>
> Key: MAPREDUCE-4751
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4751
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.3, 2.0.2-alpha
> Reporter: Ravi Prakash
> Assignee: Vinod Kumar Vavilapalli
> Attachments: MAPREDUCE-4751-20121108.txt, TaskAttemptStateGraph.jpg
>
>
> We found some jobs were stuck in KILL_WAIT for days on end. The RM shows them
> as RUNNING. When you go to the AM, it shows it in the KILL_WAIT state, and a
> few maps running. All these maps were scheduled on nodes which are now in the
> RM's Lost nodes list. The running maps are in the FAIL_CONTAINER_CLEANUP state
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
