[
https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321696#comment-15321696
]
Carlo Curino commented on YARN-5215:
------------------------------------
[~jlowe] thanks for the comment, very useful for context, and you bring up good
points on how users "perceive" the cluster.
[~elgoiri], correct me if I am wrong, but this feature seems ideal to
"scavenge" a YARN cluster out of otherwise utilized machines. In these
settings, users should be aware that the cluster is not constant, i.e., the
effects of the fluctuations are non-trivial and expected. However, I agree with
you that surfacing them in the UI somehow is important.
All in all, I see a strong connection with over-commit, but this should be
represented not just as a heavily overcommitted cluster. I agree with
[~elgoiri] that it is useful to build this feature in a way that more
explicitly acknowledges that YARN is not the only thing running on the cluster.
At the same time, we should try to have a set of configurable that makes
over/under-commit appear unified and coherent to the admins, and UIs that
surface them properly to users. [~elgoiri] since you were involved in
YARN-1011, can you propose a way to do that?
> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>
> Key: YARN-5215
> URL: https://issues.apache.org/jira/browse/YARN-5215
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Inigo Goiri
> Attachments: YARN-5215.000.patch
>
>
> Currently YARN runs containers in the servers assuming that they own all the
> resources. The proposal is to use the utilization information in the node and
> the containers to estimate how much is consumed by external processes and
> schedule based on this estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
