[
https://issues.apache.org/jira/browse/YARN-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270181#comment-16270181
]
Arun Suresh commented on YARN-2889:
-----------------------------------
Thanks for your interest in this JIRA [~cheersyang].
We have actually implemented a bit of this as part of YARN-7258. The basic idea
is that - originally, if an AM asks for x containers in an allocate call, the
{{OpportunisticContainerAllocator}} would respond with x containers. The
containers would be distributed (in a simple round-robin manner) across the top
K least loaded nodes. The problem with this is that if there are a lot of
applications running and all of them asking for O containers - or if AMs
indiscriminately ask for too many O containers in a single call, There would be
too many O containers allocated and most of them will get pre-emempted -
leading to a lot of wasted work.
In YARN-7258, we inserted a restriction that in a single allocate call, the AM
will get a max of K containers (where K is the top-K least loaded nodes - In
addition we also try to allocate a container on the same rack, if a request
node is not present in the top K - or on the least loaded node if no node from
the same rack is found in the top K). The remaining requests are queued and
retried in the next allocate call (with possibly a new top-K list - if enough
time has passed).
> Limit in the number of opportunistic container requests per AM
> --------------------------------------------------------------
>
> Key: YARN-2889
> URL: https://issues.apache.org/jira/browse/YARN-2889
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Reporter: Konstantinos Karanasos
> Assignee: Arun Suresh
>
> We introduce a way to limit the number of queueable requests that each AM can
> submit to the LocalRM.
> This way we can restrict the number of queueable containers handed out by the
> system, as well as throttle down misbehaving AMs (asking for too many
> queueable containers).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
