Karthik Kambatla commented on YARN-1011:

bq. Would it make sense to make over-subscription a global property set by the 
RM instead of per-node?
Good question. I thought about it quite some. Here is my reasoning for doing on 
the NM side. We can always switch back to defining it to the RM if that makes 
more sense.
# Even if we have the knob on the RM, the node still has to support it: monitor 
the resource usage on the node and kill the OPPORTUNISTIC containers if need 
be. On a cluster with NMs of different versions (say, during a rolling 
upgrade), the RM will have to keep track of NMs that support over-subscription. 
So, we do need some config for the NM anyway. Further, there could be 
node-specific conditions - hardware, other services running on the node etc. - 
that could affect the over-subscription capacity of the node. For instance, it 
might be okay to sign up for 90% of the advertised capacity on node A, but only 
80% on the node B. And, this ability to soak up extra work could change over 
# In terms of implementation, the node already sends its capacity and its 
aggregate-container-utilization. It might as well send an 
oversubscription-percentage over, which is interpreted as the fraction of its 
advertised capacity. e.g. A node with 64 GB memory could advertise its capacity 
as 50 GB and oversubscription-percentage 0.9. The RM could schedule upto 45 GB 
of utilization. An oversubscription-percentage <= 0 would indicate the feature 
is turned off. 

bq. What would be the first policy to implement? I guess we can define it in 
The simplest policy would likely be just assuming there are more resources on 
the node, and continue allocating with the same policies we use today for 
free/unallocated resources. 
This should work okay for the FairScheduler. I am less familiar with the 
intricate details of CS, but would think it should apply there as well. 
[~leftnoteasy] - thoughts? 

> [Umbrella] Schedule containers based on utilization of currently allocated 
> containers
> -------------------------------------------------------------------------------------
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf
> Currently RM allocates containers and assumes resources allocated are 
> utilized.
> RM can, and should, get to a point where it measures utilization of allocated 
> containers and, if appropriate, allocate more (speculative?) containers.

This message was sent by Atlassian JIRA

Reply via email to