[
https://issues.apache.org/jira/browse/YARN-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242282#comment-17242282
]
Peter Bacsko edited comment on YARN-10496 at 12/2/20, 11:48 AM:
----------------------------------------------------------------
My take: clearly I think it's between #1 and #2.
Both require re-calculating the underlying percentages, but in case of #1 it's
hidden from us because our fundamental unit is weight. However, current CS
users might be perfectly happy with the renormalization to 100 so they might
opt for #2.
But since this issue came up in the context of FS-to-CS migration, it's
probably more important to FS users. So I'd vote for approach #1, introducing
weight mode.
There's one thing we have to think about: if we start to use the "weight"
concept, it has a large effect on the UI, because (at least RM UIv2) it
displays percentages everywhere. Every part of the code where we use pct values
(logs, metrics) should be updated to reflect the setting, too. So it's
something that we have to look into.
was (Author: pbacsko):
My take: clearly I think it's between #1 and #2.
Both require re-calculating the underlying percentages, but in case of #1 it's
hidden from us because our fundamental unit is weight. However, current CS
users might be perfectly happy with the renormalization to 100 so they might
opt for #2.
But since this issue came up in the context of FS-to-CS migration, it's
probably more important to FS users. So I'd vote for approach #1, introducing
weight mode.
> [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
> ---------------------------------------------------------------------
>
> Key: YARN-10496
> URL: https://issues.apache.org/jira/browse/YARN-10496
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacity scheduler
> Reporter: Wangda Tan
> Priority: Major
>
> CapacityScheduler today doesn’t support an auto queue creation which is
> flexible enough. The current constraints:
> * Only leaf queues can be auto-created
> * A parent can only have either static queues or dynamic ones. This causes
> multiple constraints. For example:
> * It isn’t possible to have a VIP user like Alice with a static queue
> root.user.alice with 50% capacity while the other user queues (under
> root.user) are created dynamically and they share the remaining 50% of
> resources.
>
> * In comparison, FairScheduler allows the following scenarios, Capacity
> Scheduler doesn’t:
> ** This implies that there is no possibility to have both dynamically
> created and static queues at the same time under root
> * A new queue needs to be created under an existing parent, while the parent
> already has static queues
> * Nested queue mapping policy, like in the following example:
> |<rule name="nestedUserQueue" create=”true”>
> <rule name="primaryGroup" create="true" />
> </rule>|
> * Here two levels of queues may need to be created
> If an application belongs to user _alice_ (who has the primary_group of
> _engineering_), the scheduler checks whether _root.engineering_ exists, if it
> doesn’t, it’ll be created. Then scheduler checks whether
> _root.engineering.alice_ exists, and creates it if it doesn't.
>
> When we try to move users from FairScheduler to CapacityScheduler, we face
> feature gaps which blocks users migrate from FS to CS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
