Carlo Curino commented on YARN-4195:

The posted patch is a rebase of our working branch. It is useful to kickstart 
reviewing/discussion, but probably not ready to commit as-is.

In general, the patch adds the support of tracking per-label resource 
allocations in the ReservationSystem (and performs a bit of cleanups).

The one key point worth discussing is related to partitions vs labels:
 # to ensure that capacity per node label means anything both queues in the CS 
(and reservations here) are forced to refer to "partitions" of the underlying 
 # this is annoying and limiting. For example if I have GPU and PUBLICIP as my 
desired user-visible labels, I would have to manually express all 4 
NOT-GPU_NOT-PUBLICIP when defining queues, and worst, the users will be forced 
to express their needs in the same form. For example a user that only cares 
about running on {{GPU}} would have to say: {{GPU_PUBLICIP OR 
 # we propose an improvement: 
 ## Internally the system tracks partitions. 
 ## administrators configuring queues do so at the partition level
 ## users are allowed to express their job needs in terms of labels (and the 
system internally converts this into partitions)
 ## users can reserve in terms of labels (and the system internally converts 
this into partitions)

It is generally provable that an arbitrary expression of labels can be 
represented as an OR of partitions (or in disjunctive-normal-form). In this and 
future patches we have a version of this using a mix of JEXL and ad-hoc 
accelerations (JEXL was too slow in some of our tests). However, I believe  
[~chris.douglas] has a better version of this, which uses a cool algorithm 
which skips the conversion to DNF. We should plug these here.

More generally, this improvement can be of general use for CS (and in the 
future FS) to expose a nicer API to users.

(Once again, marking patch as ready to get the conversation going, some cleanup 
still required) 

> Support of node-labels in the ReservationSystem "Plan"
> ------------------------------------------------------
>                 Key: YARN-4195
>                 URL: https://issues.apache.org/jira/browse/YARN-4195
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-4195.patch
> As part of YARN-4193 we need to enhance the InMemoryPlan (and related 
> classes) to track the per-label available resources, as well as the per-label
> reservation-allocations.

This message was sent by Atlassian JIRA

Reply via email to