Craig Welch commented on YARN-2496:

ach, not finished - anyway, re 2 - a particular job may well only be able to 
use nodes with one label in a queue, and so if the headroom includes nodes 
without that label, we'll end up with another deadlock case where it spins up 
reducers too early and then can't complete its maps.  It is definately a valid 
usecase to have a queue with two lables (a and b, as in this example) and an 
app which is consistently requesting only one of those two labels (from 
submission to completion...) - perhaps only "a" nodes have the "special 
resource" it needs (special hardware capability, etc).  For this reason, 
headroom should reflect the labels in the last resource request from the 
application, not the queue's labels.

(-re 5, I though * could be in requests, if no, then should not be an issue.)

> [YARN-796] Changes for capacity scheduler to support allocate resource 
> respect labels
> -------------------------------------------------------------------------------------
>                 Key: YARN-2496
>                 URL: https://issues.apache.org/jira/browse/YARN-2496
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, 
> YARN-2496.patch
> This JIRA Includes:
> - Add/parse labels option to {{capacity-scheduler.xml}} similar to other 
> options of queue like capacity/maximum-capacity, etc.
> - Include a "default-label-expression" option in queue config, if an app 
> doesn't specify label-expression, "default-label-expression" of queue will be 
> used.
> - Check if labels can be accessed by the queue when submit an app with 
> labels-expression to queue or update ResourceRequest with label-expression
> - Check labels on NM when trying to allocate ResourceRequest on the NM with 
> label-expression
> - Respect  labels when calculate headroom/user-limit

This message was sent by Atlassian JIRA

Reply via email to