Craig Welch commented on YARN-2496:

Let's say user UserU is running two jobs, JobA and JobB, in queue QueueAB, 
which has both LabelA and LabelB.  JobA has label LabelA, JobB has LabelB.  
It's a two node cluster, Node1 and Node2.  Node1 has LabelA, Node2 has LabelB

Let's say the user has access to 100% of the cluster (just the one queue, etc).

Let's say that JobA is using 4G of Ram and JobB is also using 4G.  Let's say 
each node has 6G.

Headroom Calculation for JobA:

userConsumed = 8G
maxCapacityConsiderLabelA = 6G (Node1 only)
headroom = -2G (assume it will normalize to 0G)

However, the user should still be able to use the remaining 2G for JobA, as 
they are only using 4 of the 6G available to that label. 

The issue I see is userConsumed, as maxCapacityConsiderLabel considers the 
label, but userConsumed does not, it should only be "userConsumedForLabel", if 
it is, JobA would see 2G as it should (as the consumed for LabelA is only 4G.)  
The problem, I think, is in subtracting a cross-label value from a per-label 

> [YARN-796] Changes for capacity scheduler to support allocate resource 
> respect labels
> -------------------------------------------------------------------------------------
>                 Key: YARN-2496
>                 URL: https://issues.apache.org/jira/browse/YARN-2496
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, 
> YARN-2496.patch
> This JIRA Includes:
> - Add/parse labels option to {{capacity-scheduler.xml}} similar to other 
> options of queue like capacity/maximum-capacity, etc.
> - Include a "default-label-expression" option in queue config, if an app 
> doesn't specify label-expression, "default-label-expression" of queue will be 
> used.
> - Check if labels can be accessed by the queue when submit an app with 
> labels-expression to queue or update ResourceRequest with label-expression
> - Check labels on NM when trying to allocate ResourceRequest on the NM with 
> label-expression
> - Respect  labels when calculate headroom/user-limit

This message was sent by Atlassian JIRA

Reply via email to