Wangda Tan commented on YARN-3806:

Hi [~wshao],
Thanks for providing thoughts about this, I took a quick look at attached 
design doc, some comments (Please correct me if I missed anything).

The JIRA wants to tackle following issues
# Pluggable preemption policy
# Be able to add other allocation policies
# Application level configuration 
# Decouple application / nodes from scheduler

#1/#2, should be already covered by YARN-3306. It doesn't include detailed 
ParentQueue policy in design doc, but we plan to extend it to ParentQueue as 
mentioned in YARN-3306. And I found "preemptResource" and "acquireResource" in 
your design are very close to what we have in CS. If you have time, could you 
take a look at YARN-3318 (which is already committed), ordering policy is part 
of queue policy, is that what you were trying to do?

#3, I'm not sure if it is a valid usecase, I can understand admin can set 
"maximum-limit" of application, but setting minimum-share by app sounds not a 
fair allocation.

#4, Now we have an common abstraction of application 
(SchedulerApplicationAttempt) and node (SchedulerNode) for different scheduler 
implementations. Are you suggesting eliminate scheduler-specific implementation 
such as FiCaSchedulerApp/FiCaSchedulerNode? I think that might be problematic, 
you can think the app is also a pluggable implementation. For instance, FS/CS 
have different logic in app level, such as: how to do delayed scheduling, 
limits, etc.

To details in the design doc:
- Each LeafQueue can run at most one app, that seems not like a "queue". This 
is very restrictive, for example, if there's a requirement needs 1k app running 
at the same time, do you need to configure 1k LeafQueues? And how to choose 
where to submit application?
- YARN-2986 is trying to create a unified view of scheduler configuration, is 
there any overlap between YARN-2986 and configuration model mentioned in your 


> Proposal of Generic Scheduling Framework for YARN
> -------------------------------------------------
>                 Key: YARN-3806
>                 URL: https://issues.apache.org/jira/browse/YARN-3806
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>            Reporter: Wei Shao
>         Attachments: ProposalOfGenericSchedulingFrameworkForYARN-V1.0.pdf, 
> ProposalOfGenericSchedulingFrameworkForYARN-V1.1.pdf
> Currently, a typical YARN cluster runs many different kinds of applications: 
> production applications, ad hoc user applications, long running services and 
> so on. Different YARN scheduling policies may be suitable for different 
> applications. For example, capacity scheduling can manage production 
> applications well since application can get guaranteed resource share, fair 
> scheduling can manage ad hoc user applications well since it can enforce 
> fairness among users. However, current YARN scheduling framework doesn’t have 
> a mechanism for multiple scheduling policies work hierarchically in one 
> cluster.
> YARN-3306 talked about many issues of today’s YARN scheduling framework, and 
> proposed a per-queue policy driven framework. In detail, it supported 
> different scheduling policies for leaf queues. However, support of different 
> scheduling policies for upper level queues is not seriously considered yet. 
> A generic scheduling framework is proposed here to address these limitations. 
> It supports different policies for any queue consistently. The proposal tries 
> to solve many other issues in current YARN scheduling framework as well.

This message was sent by Atlassian JIRA

Reply via email to