[
https://issues.apache.org/jira/browse/YARN-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643316#comment-13643316
]
Sandy Ryza commented on YARN-326:
---------------------------------
Just chatted with Karthik offline about this, and I think his approach his
approach makes sense as a good hybrid. To state it a little more verbosely, a
non-fractional minimum share can be configured for each resource. A queue is
considered under its minimum share if the dominant resource in its current
allocation with respect to the cluster capacity is below its minimum share for
that resource. For a single resource, this reduces to the existing way things
work. To achieve the kind of fraction of dominant resource that Andrew speaks
about, one can simply configure shares for different resources on a queue at
the same fractions of the cluster capacity. And if we want to emphasize the
geology department's minimum share towards CPU, we can do that as well.
A concrete example would be: queue A's minimum resource vector is 10 CPU, 3 GB.
The cluster capacity is 15 CPU, 15 GB. If queue A already has allocated <4
CPU, 2 GB>, <1 CPU, 2 GB>, or <9 CPU, 8 GB>, it will be considered under its
minimum share and get a container before any queue that is not. If queue A has
<3 CPU, 4 GB> or <11 CPU, 8 GB>, it will not.
While configuring shares as fractions of the cluster size would make things
more resilient to changing clusters, I think that in order to maintain
compatibility, we would have to allow both, and this is probably work for a
different JIRA.
Is there anything we're missing here?
> Add multi-resource scheduling to the fair scheduler
> ---------------------------------------------------
>
> Key: YARN-326
> URL: https://issues.apache.org/jira/browse/YARN-326
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: scheduler
> Affects Versions: 2.0.2-alpha
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
> Attachments: FairSchedulerDRFDesignDoc.pdf, YARN-326.patch,
> YARN-326.patch
>
>
> With YARN-2 in, the capacity scheduler has the ability to schedule based on
> multiple resources, using dominant resource fairness. The fair scheduler
> should be able to do multiple resource scheduling as well, also using
> dominant resource fairness.
> More details to come on how the corner cases with fair scheduler configs such
> as min and max resources will be handled.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
