Allen Wittenauer commented on YARN-796:

bq. An important use case we can see is, some customers want to mark label on 
each node indicate which department/team the node belongs to, when a new team 
comes in and new machines added, labels may need to be changed.

You can solve this problem today by just running separate RMs.  In practice, 
however, marking nodes for specific teams in queue systems doesn't work because 
doing so assumes that the capacity never changes... i.e., nodes never fail.  
That happens all the time, of course, thus why percentages make a lot more 
sense.  If you absolutely want a fixed number of capacity, you still wouldn't 
mark specific nodes: you'd say "queue x gets y machines" with no specification 
of which nodes.

bq. And also, it is possible that the whole cluster is booked to run some huge 
batch job at 12am-2am for example. So such labels will be changed frequently.

Well, no, they won't. They'll happen exactly twice a day.  But it doesn't 
matter: you can solve this problem today too by just setting something that 
changes the queue acls at 12am and 2am via a cron job.  

bq.  For example user may want to run HBase master/region-servers in a group of 
nodes, and don't want any other tasks running in these nodes even if they have 
free resource. Our current queue configuration cannot solve such problem

... except, you guessed it: this is a solved problem today too.  You just need 
to make sure the container sizes that are requested consume the whole node.

bq. If we only have distributed configuration on each node, it is a nightmare 
for admins to re-configure.

Hi. My name is Allen and I'm an admin.  Even if using labels for doing this 
type of scheduling was sane, it still wouldn't be a nightmare because any 
competent admin would use configuration management to roll out changes to the 
nodes in a controlled manner.

But more importantly: these use cases are *solved problems* and have been in 
YARN for a very long time.

> Allow for (admin) labels on nodes and resource-requests
> -------------------------------------------------------
>                 Key: YARN-796
>                 URL: https://issues.apache.org/jira/browse/YARN-796
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Arun C Murthy
>            Assignee: Wangda Tan
>         Attachments: LabelBasedScheduling.pdf, 
> Node-labels-Requirements-Design-doc-V1.pdf, YARN-796.patch
> It will be useful for admins to specify labels for nodes. Examples of labels 
> are OS, processor architecture etc.
> We should expose these labels and allow applications to specify labels on 
> resource-requests.
> Obviously we need to support admin operations on adding/removing node labels.

This message was sent by Atlassian JIRA

Reply via email to