[
https://issues.apache.org/jira/browse/MAPREDUCE-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646183#comment-13646183
]
Carlo Curino commented on MAPREDUCE-5196:
-----------------------------------------
The attached patch presents a checkpoint-based policy. This patch captures all
the changes in the AM, and in the TaskUmbilicalProtocol needed to support
checkpoint-based task preemption.
The CheckpointAMPreemptionPolicy is designed to do the following:
1) ignores the request for preemption of maps (as maps are typically short-run
and not worth preempting).
2) it propagates requests for premption for reducers, by looking up the
taskAttemptId corresponding to the container we are asked to preempt
3) the policy also manages CheckpointIDs (i.e., references to the location of
the checkpoint) on behalf of individual tasks (on restart a task ask whether it
should start from scratch or form an existing checkpoint).
The umbilical protocol has been changed so that tasks will receive feedback
from the AM whether they should preempt. This happens both on ping and
statusUpdate (folded now in a single message type).
We also enhance the TaskAttemptImpl state machine to include the notion of
preemption.
The
> CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5196
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5196
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Attachments: MAPREDUCE-5196.patch
>
>
> This JIRA tracks a checkpoint-based AM preemption policy. The policy handles
> propagation of the preemption requests received from the RM to the
> appropriate tasks, and bookeeping of checkpoints. Actual checkpointing of the
> task state is handled in upcoming JIRAs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
