Karthik Kambatla commented on YARN-1011:

bq, Just to make sure I understand. When you say max threshold < 1 are you 
saying an NM could not advertise 48 vcores if there are only 24 vcores 
physically available?
You can continue to advertise more vcores. 

Consider a cluster with nodes of 1 physical core. Let us say each node 
advertises 10 *vcores*. Today, let us say your CPU utilization under these 
settings is 50% running 10 containers. All these containers in this context 
would be GUARANTEED containers. I am proposing we set a max threshold for the 
RM over-allocating containers to 95%.This essentially means, the RM allocates 
OPPORTUNISTIC containers on this node (that has been previously fully 
allocated) until we hit the utilization threshold of 95% - say, running 19 
containers. At this point if one container's usage goes higher taking us beyond 
95%, we kill enough OPPORTUNISTIC containers to bring this under 95%. May be, 
the max allowed threshold could be higher - 99%. I am wary of setting it to 
100% unless we have some other way of differentiating "running comfortably at 
100%" vs "contention at 100%" because both look the same.  Also, I am assuming 
people would be very happy with 95% utilization if we achieve that :)

bq. nodes can be at 100% CPU, 100% Network, or 100% Disk for long periods of 
time (several minutes). Memory could get to something like 80% before 
corrective action would be required. 
I am beginning to see the need for different thresholds for different 
resources. While I wouldn't necessarily shoot for 100, I can see someone 
configuring it to 95% CPU, 85% network (as this could spike significantly with 
shuffle etc.), 90% disk, 80% memory. And, we would stop over-allocating the 
moment we hit *any one* of these thresholds. 

Should we keep it simple to begin with and have one config, and add other 
configs in the future? Or, do you think the config-per-resource should be there 
from the get go? 

> [Umbrella] Schedule containers based on utilization of currently allocated 
> containers
> -------------------------------------------------------------------------------------
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf
> Currently RM allocates containers and assumes resources allocated are 
> utilized.
> RM can, and should, get to a point where it measures utilization of allocated 
> containers and, if appropriate, allocate more (speculative?) containers.

This message was sent by Atlassian JIRA

Reply via email to