[
https://issues.apache.org/jira/browse/MAPREDUCE-5841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977697#comment-13977697
]
Sangjin Lee commented on MAPREDUCE-5841:
----------------------------------------
I just wanted to clarify a couple of things regarding -kill-task or -fail-task
with uber jobs. Since the uber job executes task attempts serially, we can have
a situation where a mapper attempt is killed but the new mapper attempt will be
queued behind an existing reducer attempt. In that case, the job will not be
able to make progress if the reducer needs the killed mapper to finish. I think
this happens already before these changes, and what we do here is likely not
going to change that.
Also, another situation is if mapper/reducer tasks do not respond to interrupt
(i.e. uninterruptible). If a task does not respond to interrupt, -kill-task
won't necessarily work.
Outside those cases, this fix can probably improve the situation and make the
job complete successfully by interrupting the targeted attempt.
Is that the same as your understanding?
> uber job doesn't terminate on getting mapred job kill
> -----------------------------------------------------
>
> Key: MAPREDUCE-5841
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5841
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.3.0
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Attachments: mapreduce-5841.patch
>
>
> If you issue a "mapred job -kill" against a uberized job, the job (and the
> yarn application) state transitions to KILLED, but the application master
> process continues to run. The job actually runs to completion despite the
> killed status.
> This can be easily reproduced by running a sleep job:
> {noformat}
> hadoop jar hadoop-mapreduce-client-jobclient-2.3.0-tests.jar sleep -m 1 -r 0
> -mt 300000
> {noformat}
> Issue a kill with "mapred job -kill \[job-id\]". The UI will show the job
> (app) is in the KILLED state. However, you can see the application master is
> still running.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
