[jira] [Commented] (YARN-3306) [Umbrella] Proposing per-queue Policy driven scheduling in YARN

[Karthik Kambatla (JIRA)]

    [ 
https://issues.apache.org/jira/browse/YARN-3306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360697#comment-14360697
 ] 

Karthik Kambatla commented on YARN-3306:
----------------------------------------

[~cwelch] - thanks for the clarifications. I also spoke to Vinod offline on the 
goals and likely path of this work. 

I see the benefits of having a single scheduler with pluggable policies. I feel 
it might be easier to implement a new scheduler and plug-in FS and CS policies 
into it. However, I understand the iterative approach you propose will get 
validation from current users along the way.

I am a little circumspect about the iterative approach and how we avoid 
regressions, but remain hopeful the code will convince me it is the right 
approach. I would like to be involved in the work here, can we work on a branch 
and merge in as and when appropriate.  

> [Umbrella] Proposing per-queue Policy driven scheduling in YARN
> ---------------------------------------------------------------
>
>                 Key: YARN-3306
>                 URL: https://issues.apache.org/jira/browse/YARN-3306
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: scheduler
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>         Attachments: PerQueuePolicydrivenschedulinginYARN.pdf
>
>
> Scheduling layout in Apache Hadoop YARN today is very coarse grained. This 
> proposal aims at converting today’s rigid scheduling in YARN to a per­-queue 
> policy driven architecture.
> We propose the creation of a c​ommon policy framework​ and implement a​common 
> set of policies​ that administrators can pick and chose per queue
>  - Make scheduling policies configurable per queue
>  - Initially, we limit ourselves to a new type of scheduling policy that 
> determines the ordering of applications within the l​eaf ­queue
>  - In the near future, we will also pursue parent­ queue level policies and 
> potential algorithm reuse through a separate type of policies that control 
> resource limits per queue, user, application etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)