[
https://issues.apache.org/jira/browse/YARN-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588955#comment-14588955
]
Wei Shao commented on YARN-3806:
--------------------------------
Hi Wangda Tan,
Thanks for reading the proposal! Some quick replies.
1. The queue which the applications submit to will create leaf queue for each
accepted application in runtime, application won't submit to leaf queue
directly. With the concept of 'single application queue', scheduling framework
can manage queues and applications (since application is also a queue) in tree
consistently. The scheduler even doesn't need know the concept of application.
So the proposed preemption model (support preemption among applications in same
queue), resource allocation model, and configuration model (support application
level configuration) can be implemented consistently. This is one of the design
difference between YARN-3306 and this proposal.
2. Application level configuration. Yes, application specific minimalShare for
fair scheduling isn't useful in terms of fairness. However, there are some
other cases it might be useful.
a. In configuration section for the queue with fair scheduling, there could
be a template configuration for all single application queues it will create in
runtime, and minimalThreshold can be specified there. Then minimalShare for
each application is minimalThreshold*fairShare and updated at runtime.
Preemption model can preempt resources between applications according to
minimalShare while respecting fairness.
b. For the queue with capacity scheduling or scheduling policy proposed in
YARN-3807, each application certainly can have its own capacity to meeting SLA.
By having 'single application queue', all application configurations can be
respected consistently.
Regarding your other comments, I will take a look at the jiras you mentioned
and reply later. Thanks!
> Proposal of Generic Scheduling Framework for YARN
> -------------------------------------------------
>
> Key: YARN-3806
> URL: https://issues.apache.org/jira/browse/YARN-3806
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
> Reporter: Wei Shao
> Attachments: ProposalOfGenericSchedulingFrameworkForYARN-V1.0.pdf,
> ProposalOfGenericSchedulingFrameworkForYARN-V1.1.pdf
>
>
> Currently, a typical YARN cluster runs many different kinds of applications:
> production applications, ad hoc user applications, long running services and
> so on. Different YARN scheduling policies may be suitable for different
> applications. For example, capacity scheduling can manage production
> applications well since application can get guaranteed resource share, fair
> scheduling can manage ad hoc user applications well since it can enforce
> fairness among users. However, current YARN scheduling framework doesn’t have
> a mechanism for multiple scheduling policies work hierarchically in one
> cluster.
> YARN-3306 talked about many issues of today’s YARN scheduling framework, and
> proposed a per-queue policy driven framework. In detail, it supported
> different scheduling policies for leaf queues. However, support of different
> scheduling policies for upper level queues is not seriously considered yet.
> A generic scheduling framework is proposed here to address these limitations.
> It supports different policies for any queue consistently. The proposal tries
> to solve many other issues in current YARN scheduling framework as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
