[
https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322685#comment-15322685
]
Jason Lowe commented on YARN-5215:
----------------------------------
bq. All in all, I see a strong connection with over-commit, but this should be
represented not just as a heavily overcommitted cluster.
I guess I just see it differently. As you mentioned, YARN is "scavanging"
resources from a cluster that is doing something else. In that sense YARN is
stealing underutilized resources, and that's exactly what overcommit does. A
user that thinks they have full use of the node but it can be taken away
arbitrarily by external load is just fooling themselves into a false sense of
guaranteed capacity. They really are using a heavily overcommitted cluster in
practice, so why shouldn't YARN reflect that reality?
bq. I was thinking on exposing the getExternalUtilization() or the updated
getUnallocated() through the Web UI, etc.
As I mentioned above, that alone is not going to update total cluster capacity
nor the capacity available in various scheduler queues. Users will have to do
mental math with that metric and the reported cluster available capacity to
understand why a scheduler queue showing free resources refuses to schedule
more containers. In addition the reduced capacity will not be properly
accounted for among the scheduler queues, so the scheduler will end up
scheduling differently than one that was aware of the true cluster capacity.
> Scheduling containers based on external load in the servers
> -----------------------------------------------------------
>
> Key: YARN-5215
> URL: https://issues.apache.org/jira/browse/YARN-5215
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Inigo Goiri
> Attachments: YARN-5215.000.patch, YARN-5215.001.patch
>
>
> Currently YARN runs containers in the servers assuming that they own all the
> resources. The proposal is to use the utilization information in the node and
> the containers to estimate how much is consumed by external processes and
> schedule based on this estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
