Wangda Tan created YARN-10496:
---------------------------------
Summary: [Umbrella] Support Flexible Auto Queue Creation in
Capacity Scheduler
Key: YARN-10496
URL: https://issues.apache.org/jira/browse/YARN-10496
Project: Hadoop YARN
Issue Type: New Feature
Components: capacity scheduler
Reporter: Wangda Tan
CapacityScheduler today doesn’t support an auto queue creation which is
flexible enough. The current constraints:
* Only leaf queues can be auto-created
* A parent can only have either static queues or dynamic ones. This causes
multiple constraints. For example:
* It isn’t possible to have a VIP user like Alice with a static queue
root.user.alice with 50% capacity while the other user queues (under root.user)
are created dynamically and they share the remaining 50% of resources.
* In comparison, FairScheduler allows the following scenarios, Capacity
Scheduler doesn’t:
** This implies that there is no possibility to have both dynamically created
and static queues at the same time under root
* A new queue needs to be created under an existing parent, while the parent
already has static queues
* Nested queue mapping policy, like in the following example:
|<rule name="nestedUserQueue" create=”true”>
<rule name="primaryGroup" create="true" />
</rule>|
* Here two levels of queues may need to be created
If an application belongs to user _alice_ (who has the primary_group of
_engineering_), the scheduler checks whether _root.engineering_ exists, if it
doesn’t, it’ll be created. Then scheduler checks whether
_root.engineering.alice_ exists, and creates it if it doesn't.
When we try to move users from FairScheduler to CapacityScheduler, we face
feature gaps which blocks users migrate from FS to CS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
