[jira] [Commented] (YARN-3306) [Umbrella] Proposing per-queue Policy driven scheduling in YARN

[Karthik Kambatla (JIRA)]

    [ 
https://issues.apache.org/jira/browse/YARN-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355902#comment-14355902
 ] 

Karthik Kambatla commented on YARN-3306:
----------------------------------------

I agree with the pain points of a fragmented scheduler state in YARN. 

Is the proposal only to add a policy for assigning resources in the leaf queue 
and leave the rest of the schedulers as is? If so, can you elaborate the 
advantages (usecases)? 

Or, is the proposal to arrive at a single scheduler implementation with the 
existing schedulers becoming just policies? If so, we might want to implement a 
new scheduler, may be in a new branch, and capture all the existing features 
and phase the two out. 

> [Umbrella] Proposing per-queue Policy driven scheduling in YARN
> ---------------------------------------------------------------
>
>                 Key: YARN-3306
>                 URL: https://issues.apache.org/jira/browse/YARN-3306
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: scheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: PerQueuePolicydrivenschedulinginYARN.pdf
>
>
> Scheduling layout in Apache Hadoop YARN today is very coarse grained. This 
> proposal aims at converting today’s rigid scheduling in YARN to a per­-queue 
> policy driven architecture.
> We propose the creation of a c​ommon policy framework​ and implement a​common 
> set of policies​ that administrators can pick and chose per queue
>  - Make scheduling policies configurable per queue
>  - Initially, we limit ourselves to a new type of scheduling policy that 
> determines the ordering of applications within the l​eaf ­queue
>  - In the near future, we will also pursue parent­ queue level policies and 
> potential algorithm reuse through a separate type of policies that control 
> resource limits per queue, user, application etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)