[
https://issues.apache.org/jira/browse/YARN-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14379092#comment-14379092
]
Wangda Tan commented on YARN-3214:
----------------------------------
Hi [~lohit],
Thanks for review the doc, for your comments:
bq. If so, it would become too restrictive. Labels on nodes can be seen in
multiple dimension (from app's resource, machine resource and also usecase
resouce, eg backfill jobs are placed on specific set of nodes). In those cases
we should have ability to have multiple labels on node
Yes, now we only support one label for each node (partition). We temporarily
support only one for each node is, if we have multiple labels on each node, it
will hard to do resource planning (like what we did, we can say queue-A can use
40% of label-X and queue-B can use 60% of label-X). Assume a node with label-X
and label-Y, and its resource is 10G, it will be hard to say the node has 10G
resource of (X+Y) OR 10G resource of X and Y. This also makes preemption hard
to do. A tradeoff is, if we don't plan resource share (or capacity) on
node-labels, some resource could be wasted and queues can be starved when they
still under their configured capacity.
Multiple labels on node (we call this constraint) is in design stage, we've
some thoughts about it, and will push it to community once it get a better
share -- should not take too long.
bq. Also, in the documents there is mention of scheduling apps without any
labels being scheduled on labeled nodes if resources are idle. Does that also
cover apps which could have different label other than A/B, but still have a
label be placed on these nodes when there is free resources available?
No, it will only try to allocate non-labeled requests to labeled nodes, if a
resource request explicitly asks node label, we will only allocate
corresponding labeled resource for it.
> Add non-exclusive node labels
> ------------------------------
>
> Key: YARN-3214
> URL: https://issues.apache.org/jira/browse/YARN-3214
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler, resourcemanager
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Attachments: Non-exclusive-Node-Partition-Design.pdf
>
>
> Currently node labels partition the cluster to some sub-clusters so resources
> cannot be shared between partitioned cluster.
> With the current implementation of node labels we cannot use the cluster
> optimally and the throughput of the cluster will suffer.
> We are proposing adding non-exclusive node labels:
> 1. Labeled apps get the preference on Labeled nodes
> 2. If there is no ask for labeled resources we can assign those nodes to non
> labeled apps
> 3. If there is any future ask for those resources , we will preempt the non
> labeled apps and give them back to labeled apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
