Hi Lei, This is a very interesting topic, we have LOTS OF discussions with many folks for resource vs. label.
The major concerns of not accepting user-provided resource type to resource vectors are: - It may simply doesn't work, dominate resource fairness cannot deal with nodes with different set of resource vectors (some nodes have GPU and some not). - You cannot enforce it. For example, a node has 8 GPUs, and you allocate to 5 processes, one gets 5 and the other one gets 3. But from YARN, you cannot ensure nobody uses more than allocated. - Adding new vectors needs lots of refactoring in protocol, scheduler, etc. in YARN which is not figured out yet. Currently, YARN doesn't have plan in short term accept user-provided resource type to resource vector. But adding controllable resources like IO/bandwidth, etc is on the road map. So in the short term, for such non controllable resource, you can put them to node label, you can enforce resource sharing via partition, or just a binding requirement via attribute. Thoughts? WAngda On Thu, Jun 25, 2015 at 1:27 PM, Lei Guo <[email protected]> wrote: > Hi, Wangda, > > Actually, it's the two types of node label causing the confusion for me. > For a scheduling system, it's to find the best match between "resource" and > "workload" via different "policies". Both node partition and node > constraints are scheduling policies. The node label should be part of > resource (as an attribute), and be used in two different policies. If we > position the node label this way, there is only one type of node label. > This is related to how we define/abstract "resource" in Yarn. Currently, > only CPU/memory (extending with disk/network bandwidth) considered as > resource for a node. As you mentioned in Summit slides, GPU could also be a > resource. These are all consumable resources/attributes provided by node > while node label is non-consumable resource/attribute provided by a node. > > Lei > > -----Original Message----- > From: Wangda Tan [mailto:[email protected]] > Sent: Thursday, June 25, 2015 3:51 PM > To: [email protected] > Subject: Re: Node Label, partition or an attribute of node? > > Hi Lei, > To clarify, node label has two types, one is node partition, which is > supported already. Another is node constraint (you can also call it node > attribute), which is still under design. Both of them are string typed. > > And node label (including partition and attribute), can be either > per-application level and per-container level, this is supported by API > already. > > And you can take a look at my summit slides: > http://www.slideshare.net/Hadoop_Summit/node-labels-in-yarn-49792443. > Which > may help you better understand partition / constraint and relationship > between them. > > Thanks, > Wangda > > On Thu, Jun 25, 2015 at 12:34 PM, Lei Guo <[email protected]> wrote: > > > The definition of label in YARN-796 is pretty clear, > > > > * Node Label label that describes a node. Node can have multiple > > labels > > > > * Label expression logical combination of labels (using && and, > > || or, ! not) > > With this definition (without considering the implementation), a node > > label could be treated as a Boolean type attribute of a node. Assuming > the > > administrator define label1, label2, label3 in the system, and associate > > node1 with "label1, label2", this means the value of attribute label1 and > > label2 for node1 is true, while the value of attribute label3 will be > false > > for node1. > > > > In current Yarn implementation, seems the resource partitioning has been > > treated as the primary use case for node label, and some > > design/implementation of node label mainly consider on the partitioning > > case, even the multiple label support has been disabled in YARN-2694. To > > cover the workload resource request use case, YARN-3409 was proposed > while > > the description/example of constraint node label in YARN-3409 seems more > > like a String type attribute for a node instead of Boolean type anymore. > I > > got confused about node label. > > > > From my view, node label is one attribute of a node, and it could be used > > for different scheduling scenarios. No matter resource partitioning or > job > > scheduling constraints, a node label is just a label on a node. For > > partitioning, we can have one object calling partition or resource group > > representing the group of nodes with certain label; for workload > scheduling > > constraints purpose, we may want to extend the current application level > > constraint to be container level constraint, for one application, the > > master may request one type of resource while the slave request another > > type of resource. > > > > Comments? > > > > Lei Guo > > Senior Architect, Huawei Canada Research Centre > > > > >
