[
https://issues.apache.org/jira/browse/YARN-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270591#comment-16270591
]
Weiwei Yang commented on YARN-2889:
-----------------------------------
Thanks [~asuresh], [~kkaranasos] for the quick response.
I don't think you can expect user to respect this limit, at least not for the
first time. On the other hand, number of container requests is vary against the
size of the job, how you gonna define this limit? Therefore, it doesn't seem to
be practical to me.
Some other thoughts might be related,
# If NM queue is full, can we avoid assigning any O containers to that node?
That means when preparing top K least loaded nodes, we need to exclude nodes
whose queue is already full.
# Each queue size is limited, so I don't see why lots of O containers would
flood the system.
# You can't say an AM is malicious if it requests only opportunistic containers
(too many). Unless this was the design, then you need to setup correct user
expectation with some document, and explain what is the correct user case.
Well these comments are based on my current understanding by reading JIRA
comments and design doc, please correct me if anything goes wrong. Thanks!
> Limit in the number of opportunistic container requests per AM
> --------------------------------------------------------------
>
> Key: YARN-2889
> URL: https://issues.apache.org/jira/browse/YARN-2889
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Reporter: Konstantinos Karanasos
> Assignee: Arun Suresh
>
> We introduce a way to limit the number of queueable requests that each AM can
> submit to the LocalRM.
> This way we can restrict the number of queueable containers handed out by the
> system, as well as throttle down misbehaving AMs (asking for too many
> queueable containers).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
