[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989948#comment-16989948
]
Manikandan R commented on YARN-9892:
------------------------------------
Approach:
1. Create a new DominantResourceFairnessPolicy class constructor
2. In above class itself, Create a new DominantResourceFairnessComparator
inner class implements Comparator<SchedulableEntity> with compare() method
containing below 3-5 steps
3. Get pending resources of both Schedulable entities using
SchedulableEntity.getSchedulingResourceUsage().getPending().
4. Get available resource of corresponding queue in which these entities are
running (or Should we use cluster resource itself? I don't think so).
5. Call DominantResourceCalculator.compare() with #3 and #4 as inputs to find
the diff inside DominantResourceFairnessComparator#compare() and return the
same.
6. Similar to how comparator has been passed in other ordering policies
constructors, new comparator created in #2 can also be passed in new
DominantResourceFairnessPolicy class constructor.
While working on this, observed lot of synchronised blocks in
{{AbstractComparatorOrderingPolicy}} to operate {{entitiesToReorder}} map. I
think those places can be improved using fine grained locks as this map
operations falls on the important code path. Can we create a separate Jira for
this?
cc: [~sunilg] [~leftnoteasy] [~epayne]. Thoughts?
> Capacity scheduler: support DRF ordering policy on queue level
> --------------------------------------------------------------
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Peter Bacsko
> Assignee: Manikandan R
> Priority: Major
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity.<queue-path>.ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
