Carlo Curino commented on YARN-1707:


Thanks for the great feedback. You spot a bunch of oddities that were there due 
to previous versions of the reservation system, 
but not needed anymore, I think the updated version is definitely cleaner. 

We address 1,2 by:
 * moving addQueue and removeQueue to PlanQueue (as they were only invoked on 
instances of the subclass).
 * making uniform checks from within the PlanQueue for capacity > 0, and throw 
uniform SchedulerConfigEditException
 * fixing the log, and making the logs more uniform

We address 3,6 by:
 * merge addCapacity and subtractCapacity into a single changeCapacity
 * make checks of range limits 0,1 
(this reduced code both in CS and ReservationQueue... good call!)

We address 4 by:
 * getReservableQueues() has been renamed to getPlanQueues()

Regarding 5: ReservationQueue#getQueueName 
 * This is the result of our previous conversations with Vinod, Bikas, and 
Arun. The idea is that the user should not be 
aware of the fact that we use queues to implement reservations, and thus it 
shouldn't see the name of the reservation 
queue to be listed in the UI, but rather the name of the parent PlanQueue. More 
precisely, we have options for the UI 
to show or not the subqueues, but this differentiation is needed here to allow 
that: getQueueName for a ReservationQueue
return the parent, while getReservationQueueName() returns the actual local 

Regarding 7: DynamicQueueConf
* We currently are only dynamically assigning capacity, but you can imagine in 
the future that this is extended to set many
more parameters for a queue (user-limit factors, max applications, etc..). The 
conf-based mechanism is future-proofing against this.

Regarding 8:  ParentQueue#setChildQueues
* I don't understand the comment. This check is automatically bypassed for 
PlanQueue (that by design have no children see 
CapacityScheduler near line 562).

We are testing the new version of the patch now, and will post patch soon.

> Making the CapacityScheduler more dynamic
> -----------------------------------------
>                 Key: YARN-1707
>                 URL: https://issues.apache.org/jira/browse/YARN-1707
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>              Labels: capacity-scheduler
>         Attachments: YARN-1707.2.patch, YARN-1707.3.patch, YARN-1707.patch
> The CapacityScheduler is a rather static at the moment, and refreshqueue 
> provides a rather heavy-handed way to reconfigure it. Moving towards 
> long-running services (tracked in YARN-896) and to enable more advanced 
> admission control and resource parcelling we need to make the 
> CapacityScheduler more dynamic. This is instrumental to the umbrella jira 
> YARN-1051.
> Concretely this require the following changes:
> * create queues dynamically
> * destroy queues dynamically
> * dynamically change queue parameters (e.g., capacity) 
> * modify refreshqueue validation to enforce sum(child.getCapacity())<= 100% 
> instead of ==100%
> We limit this to LeafQueues. 

This message was sent by Atlassian JIRA

Reply via email to