[
https://issues.apache.org/jira/browse/YARN-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644327#comment-13644327
]
Bikas Saha commented on YARN-45:
--------------------------------
My understanding is the the containers being presented in PreemptionMessage are
going to be preempted by the RM some time in the near future if the RM cannot
find free resources elsewhere. The AM's are not supposed to preempt the
containers but they are encourage to checkpoint and save work. The RM can
always choose to not preempt these containers and so it would be sub-optimal
for the AM to kill these containers.
If we want to add additional information besides the set of
containers-to-be-preempted then I would prefer ResourceRequest (like it was in
the original patch) and not Resource. Not only is that symmetric but also
allows the RM to provide additional information about where to free containers.
A smarter RM could potentially ask for resources to be preempted where the
under-allocated job wants it and a smart AM could help out by choosing
containers close to the desired locations. Secondly, Resource is too amorphous
by itself. Asking an AM to free 50GB does not tell it whether the RM needs
10*5 or 50*1. Without that information the AM can end up freeing containers in
a manner that does not help the RM to meet the request of the under-allocated
job, thus failing to meet quota and wasting work at the same time.
> Scheduler feedback to AM to release containers
> ----------------------------------------------
>
> Key: YARN-45
> URL: https://issues.apache.org/jira/browse/YARN-45
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Reporter: Chris Douglas
> Assignee: Carlo Curino
> Attachments: YARN-45.patch, YARN-45.patch, YARN-45.patch,
> YARN-45.patch, YARN-45.patch, YARN-45_summary_of_alternatives.pdf
>
>
> The ResourceManager strikes a balance between cluster utilization and strict
> enforcement of resource invariants in the cluster. Individual allocations of
> containers must be reclaimed- or reserved- to restore the global invariants
> when cluster load shifts. In some cases, the ApplicationMaster can respond to
> fluctuations in resource availability without losing the work already
> completed by that task (MAPREDUCE-4584). Supplying it with this information
> would be helpful for overall cluster utilization [1]. To this end, we want to
> establish a protocol for the RM to ask the AM to release containers.
> [1] http://research.yahoo.com/files/yl-2012-003.pdf
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
