[
https://issues.apache.org/jira/browse/YARN-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643444#comment-13643444
]
Carlo Curino commented on YARN-45:
----------------------------------
We modified the patch to account for the most recent round of comments from
[~tgraves], [~kkambatl], [~vinodkv] and [~bikassaha].
In particular:
# various javadoc fixes (including @link notation)
# PreemptMessage -> PreemptionMessage
# set/get version of PreemptionMessage (and propagate through the depending
patches)
# clarified in AllocateResponse javadoc that PreemptMessage could be repeated
over time.
[~bikassaha], you are right it is likely that there will be repetitions in the
ask over time (4 above). In fact, by design the RM will "sustain" its asks
until either: 1) the need for those resources is gone, 2) the containers are
released (natural or AM-initiated completion), or 3) a timeout expires and the
RM force-kill the containers. The possible overlap among subsequent messages is
not a big concern on the AM side given our choice of a Set<ContainerID> based
PreemptionMessage. Duplicates are trivial to detect, and/or the AM can simply
implement preemption in an idempotent way (which is what we do in our mapreduce
solution).
Regarding time, in the basic implementation we have for mapreduce, the AM does
not attempt complex speculations on when to preempt, it simply acts on the
requests in a idempotent way as soon as they are received (this also maximizes
the chance to complete a checkpoint before being killed). In our design we
pushed this in an AMPreemptionPolicy, so you can easily imagine more advanced
policies to track containers over-time and speculate on when is best to
preempt. Adding more sophisticated "timing" information in the protocol is also
something I can see being an interesting addition, but I would want to spend
some more time (no pun intended) working with it, before proposing a public
protocol change---again we mention this in the attached summary document.
We get into a bit more details in the attached document, which reports as
[~vinodkv] asked a summary of the conversations around various alternatives
using resource-based specification, and about adding time.
> Scheduler feedback to AM to release containers
> ----------------------------------------------
>
> Key: YARN-45
> URL: https://issues.apache.org/jira/browse/YARN-45
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Reporter: Chris Douglas
> Assignee: Carlo Curino
> Attachments: YARN-45.patch, YARN-45.patch, YARN-45.patch
>
>
> The ResourceManager strikes a balance between cluster utilization and strict
> enforcement of resource invariants in the cluster. Individual allocations of
> containers must be reclaimed- or reserved- to restore the global invariants
> when cluster load shifts. In some cases, the ApplicationMaster can respond to
> fluctuations in resource availability without losing the work already
> completed by that task (MAPREDUCE-4584). Supplying it with this information
> would be helpful for overall cluster utilization [1]. To this end, we want to
> establish a protocol for the RM to ask the AM to release containers.
> [1] http://research.yahoo.com/files/yl-2012-003.pdf
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
