Carlo Curino commented on YARN-1712:

Attaching an interface, implementation and tests for the PlanFollower. 

The key idea is for this component to keep one (or more) Plan(s) in sync with 
the underlying scheduler. The implementation we present targets schedulers like 
Capacity and Fair schedulers that have a notion of queues and "entitlement" 
(term used to refer to both capacity and fair-scheduler weights). As per 
discussions with [~acmurthy] an alternative implementations could map plan's 
allocations to dynamically updated priorities within a single queue.

The reservation in our Plan are published to the scheduler by 
creating/destroying/resizing leaf queues (a special new kind of dynamically 
creatable/destroyable leaf queues, that are not required to be in sync with the 
config files). The policy is invoked on a timer, and makes a stateless 
comparison of scheduler queue states and plan current allocations, and compute 
deltas to determine what queues should be added, removed, resized. At the end 
of its execution the set of queues existing in the scheduler (under the 
ParentQueue corresponding to each Plan) correspond to the set of allocations 
active in the current instant in time, and the queue capacity correspond to the 
allocation size.

We operate in a stateless fashion (i.e., computing deltas on the fly, and not 
as play forward of an action log) so that we are resilient to various timing / 
queueing effects. If we are too slow or for whatever reason we are missing a 
time step, we will simply resync onto the next moment in time (or whenever we 
are invoked).

Conceptually this piece also provide a translation between reservation 
allocations, which are all expressed in "absolute" terms to queue 
capacity/weight configurations, which are all relative to current cluster 
capacity. This allows us to amortize fluctuations in cluster capacity (up to a 
point, beyond which the plan need to be reorganized). 

Like for other sub-patches of YARN-1051 I am marking this as patch-available to 
signal the fact that is ready to be reviewed, however the patch does NOT 
compile on its own.

> Admission Control: plan follower
> --------------------------------
>                 Key: YARN-1712
>                 URL: https://issues.apache.org/jira/browse/YARN-1712
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler, resourcemanager
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-1712.patch
> This JIRA tracks a thread that continuously propagates the current state of 
> an inventory subsystem to the scheduler. As the inventory subsystem store the 
> "plan" of how the resources should be subdivided, the work we propose in this 
> JIRA realizes such plan by dynamically instructing the CapacityScheduler to 
> add/remove/resize queues to follow the plan.

This message was sent by Atlassian JIRA

Reply via email to