Wei Shao commented on YARN-3806:

Hi Wangda Tan,

Thanks for reading the proposal! Some quick replies.

1. The queue which the applications submit to will create leaf queue for each 
accepted application in runtime, application won't submit to leaf queue 
directly. With the concept of 'single application queue', scheduling framework 
can manage queues and applications (since application is also a queue) in tree 
consistently. The scheduler even doesn't need know the concept of application. 
So the proposed preemption model (support preemption among applications in same 
queue), resource allocation model, and configuration model (support application 
level configuration) can be implemented consistently. This is one of the design 
difference between YARN-3306 and this proposal.

2. Application level configuration. Yes, application specific minimalShare for 
fair scheduling isn't useful in terms of fairness. However, there are some 
other cases it might be useful.
   a. In configuration section for the queue with fair scheduling, there could 
be a template configuration for all single application queues it will create in 
runtime, and minimalThreshold can be specified there. Then minimalShare for 
each application is minimalThreshold*fairShare and updated at runtime. 
Preemption model can preempt resources between applications according to 
minimalShare while respecting fairness.
   b. For the queue with capacity scheduling or scheduling policy proposed in 
YARN-3807, each application certainly can have its own capacity to meeting SLA.
By having 'single application queue', all application configurations can be 
respected consistently.

Regarding your other comments, I will take a look at the jiras you mentioned 
and reply later. Thanks!

> Proposal of Generic Scheduling Framework for YARN
> -------------------------------------------------
>                 Key: YARN-3806
>                 URL: https://issues.apache.org/jira/browse/YARN-3806
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Wei Shao
>         Attachments: ProposalOfGenericSchedulingFrameworkForYARN-V1.0.pdf, 
> ProposalOfGenericSchedulingFrameworkForYARN-V1.1.pdf
> Currently, a typical YARN cluster runs many different kinds of applications: 
> production applications, ad hoc user applications, long running services and 
> so on. Different YARN scheduling policies may be suitable for different 
> applications. For example, capacity scheduling can manage production 
> applications well since application can get guaranteed resource share, fair 
> scheduling can manage ad hoc user applications well since it can enforce 
> fairness among users. However, current YARN scheduling framework doesn’t have 
> a mechanism for multiple scheduling policies work hierarchically in one 
> cluster.
> YARN-3306 talked about many issues of today’s YARN scheduling framework, and 
> proposed a per-queue policy driven framework. In detail, it supported 
> different scheduling policies for leaf queues. However, support of different 
> scheduling policies for upper level queues is not seriously considered yet. 
> A generic scheduling framework is proposed here to address these limitations. 
> It supports different policies for any queue consistently. The proposal tries 
> to solve many other issues in current YARN scheduling framework as well.

This message was sent by Atlassian JIRA

Reply via email to