[
https://issues.apache.org/jira/browse/YARN-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590061#comment-14590061
]
Wei Shao commented on YARN-3806:
--------------------------------
The original motivation for this proposal is, in Rocket Fuel, we want to
develop and evaluate two new scheduling policies YARN-3807 & YARN-3808.
However, current yarn doesn't support per queue policy, and YARN-3306 which is
working in progress may not match all requirements of these new policies. For
example, YARN-3808 requires that the queue can create sub queue for users &
applications dynamically with specific configurations, YARN-3807 requires that
the queue can scale up/down dynamically and applications can have own
configurations.
> Proposal of Generic Scheduling Framework for YARN
> -------------------------------------------------
>
> Key: YARN-3806
> URL: https://issues.apache.org/jira/browse/YARN-3806
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
> Reporter: Wei Shao
> Attachments: ProposalOfGenericSchedulingFrameworkForYARN-V1.0.pdf,
> ProposalOfGenericSchedulingFrameworkForYARN-V1.1.pdf
>
>
> Currently, a typical YARN cluster runs many different kinds of applications:
> production applications, ad hoc user applications, long running services and
> so on. Different YARN scheduling policies may be suitable for different
> applications. For example, capacity scheduling can manage production
> applications well since application can get guaranteed resource share, fair
> scheduling can manage ad hoc user applications well since it can enforce
> fairness among users. However, current YARN scheduling framework doesn’t have
> a mechanism for multiple scheduling policies work hierarchically in one
> cluster.
> YARN-3306 talked about many issues of today’s YARN scheduling framework, and
> proposed a per-queue policy driven framework. In detail, it supported
> different scheduling policies for leaf queues. However, support of different
> scheduling policies for upper level queues is not seriously considered yet.
> A generic scheduling framework is proposed here to address these limitations.
> It supports different policies for any queue consistently. The proposal tries
> to solve many other issues in current YARN scheduling framework as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
