[
https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365259#comment-15365259
]
Inigo Goiri commented on YARN-5215:
-----------------------------------
In our internal deployment, we always reserve a buffer for the external load to
spike. This is set by tuning the available cores and memory.
[~jlowe], as you mention, we internally have preemption at both RM and NM
level. We only enable the one at NM level as it's the one with the best latency
and we don't have a need for the RM level one. As I mention in a previous
comment, this patch it's just to do scheduling in the RM, if we want to go with
the full solution, we would need:
* Schedule containers considering external load in the RM
* Expose external load in the UI
* Use history to smooth external load
* Preempting containers from the RM based on external load
* Preempting containers from the NM based on external load
> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>
> Key: YARN-5215
> URL: https://issues.apache.org/jira/browse/YARN-5215
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Inigo Goiri
> Attachments: YARN-5215.000.patch, YARN-5215.001.patch
>
>
> Currently YARN runs containers in the servers assuming that they own all the
> resources. The proposal is to use the utilization information in the node and
> the containers to estimate how much is consumed by external processes and
> schedule based on this estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
