[
https://issues.apache.org/jira/browse/MAPREDUCE-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643496#comment-13643496
]
Carlo Curino commented on MAPREDUCE-5189:
-----------------------------------------
In this patch, we show how the mapreduce AM is modified to handle requests for
preemption. This is a simple implementation of the AM-side per YARN-45 protocol
change.
We introduce minimum wiring in the RMContainerAllocator to receive the
preemption requests (as PreemptionMessage), and invoke a local
AMPreemptionPolicy in charge to decide what to do with the preemption request.
In this patch, we introduce two very simple example policies:
# NoopAMPreemptionPolicy, this policy ignores every request for preemption (and
makes "a run for it"). This is equivalent to "turning off" preemption in the
AM, and it is the default policy (for now).
# KillAMPreemptionPolicy, this policy immediately kills the task running on the
containers the RM is asking back
Both of this policies are overly simplistic, yet they are both compatible with
the proposed extension of FairScheduler and CapacityScheduler tracked in
YARN-567, YARN-568, and YARN-569.
By the design the AMPreemptionPolicy API is rather general and it should allows
us to implement more sophisticated policies with no API changes. Opinions on
this?
Related to this: with MAPREDUCE-5176 we start introducing the necessary
concepts to support a more effective policy that leverages checkpointing as a
mechanism to save the state of the computation and preempt in a work-preserving
fashion. This will be associated with a more patches detailing a more advanced
AMPreemptionPolicy and the underlying checkpoint mechanics.
> Basic AM changes to support preemption requests (per YARN-45)
> -------------------------------------------------------------
>
> Key: MAPREDUCE-5189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5189
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Reporter: Carlo Curino
> Assignee: Carlo Curino
>
> This JIRA tracks the minimum amount of changes necessary in the mapreduce AM
> to receive preemption requests (per YARN-45) and invoke a local policy that
> manages preemption. (advanced policies and mechanisms will be tracked
> separately)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
