[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016160#comment-17016160
]
Wangda Tan commented on YARN-9892:
----------------------------------
Spent a bit time to look at the code, not fully dig into all details, but I
think there's significant different between FS and CS to handle resource
calculator:
In CS, resource calculator is initialized and used across all the logic.
In FS, there's a separate SchedulingPolicy, which is a wrapper of resource
calculator. It covers areas like compute shares, sort apps/queues, compute
headrooms, etc.
For the patch uploaded in the Jira, it handles a small area only, which is sort
apps.
I want to hear thoughts from [~wilfreds] that why the drf for different queues
is a P0 feature, to me DRF is just an natural extension of Fair (which
considers multiple resource types). I still think we should optimize the logic
and knobs to the users. Cluster admin has to re-tune lots of queue capacities
and SLAs after conversion, we cannot maintain all the different behaviors.
And since there's still a global configuration of ResourceCalculator, which is
used by
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils#getNormalizedResource
when the request came in to the scheduler. I'm a bit confused that
SchedulingPolicy could conflict with the global ResourceCalculator.
> Capacity scheduler: support DRF ordering policy on queue level
> --------------------------------------------------------------
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Peter Bacsko
> Assignee: Manikandan R
> Priority: Major
> Attachments: YARN-9892-003.patch, YARN-9892.001.patch,
> YARN-9892.002.patch
>
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity.<queue-path>.ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
