[
https://issues.apache.org/jira/browse/HADOOP-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610506#action_12610506
]
Chris Smith commented on HADOOP-3687:
-------------------------------------
I agree that suspending/resuming reducers makes more sense than map tasks. I've
only implemented the pause logic in reducers at the minute, and (assuming that
it works well enough) I doubt there will be any need to add it to mappers.
As for keeping tasks in memory, it's obviously nowhere near ideal, but I view
this more as an interim solution until HADOOP-91 is resolved; I'm thinking that
with a bit of end-user diligence (ensuring that all high(er) priority jobs were
queued before the low(er) one is paused) we could probably get away with having
at most one paused task per node. Whether or not that will be problematic wrt
memory usage obviously depends on the tasks/hardware/config, and is something
I'll have to look at. If keeping them in memory proves not to be feasible then
I'll obviously have to concentrate on suspending to the [d]fs.
Owen: unfortunately a command that results in tasks being killed wouldn't
really have the effect I'm after; we don't want to waste the work that's been
done by the (possibly long-running) lower-priority tasks. At present the tasks
are killed manually to make way for the high-priority job — the idea of pausing
is to be able to preserve the progress that's already been made on those tasks
until the high-pri job is out of the way.
> Ability to pause/resume tasks
> -----------------------------
>
> Key: HADOOP-3687
> URL: https://issues.apache.org/jira/browse/HADOOP-3687
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Chris Smith
> Priority: Minor
>
> It would be nice to be able to pause (and subsequently resume) tasks that are
> currently running, in order to allow tasks from higher priority jobs to
> execute. At present it is quite easy for long-running tasks from low priority
> jobs to block a task from a newer high priority job, and there is no way to
> force the execution of the high priority task without killing the low
> priority jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
