[
https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144580#comment-16144580
]
Wangda Tan commented on YARN-7117:
----------------------------------
Discussed with [~clayb]/[~sunilg]l/[~vinodkv] offline (Thanks Clay for sharing
internal use cases). Here is our initial proposal to get more thoughts:
- Parent queue can be marked to allow auto creation of leaf queues.
(such as {{prefix.<queue-path>.auto-queue-creation.enabled}}, default is off).
We allow no sub queue specified for such parent queue.
- Minimum resource could be specified for queues which are automatically
created. (such as
{{prefix.<queue-path>.auto-queue-creation.subqueue-minimum-resource}}). After
YARN-5881, absolute resources can be specified for auto created queues.
- CS treats automatically created queues no different from normal queues, which
means scheduler will use existing logic to do preemption / fairness allocation
/ queue-ordering / user-limit, etc. for auto-created queue.
- ACL of created queue should be determined by policy. For example, if we
expect create different queue for different user, admin may set
{{prefix.<queue-path>.auto-queue-creation.admin-acl-policy=user-name-equals-to-queue-name}}.
- Auto-create queue flag can be specified in queue-mapping policy, default is
off.
A related issue (maybe it's better to discuss on a separate JIRA) is: it's
possible that queues are created but not actively used, so we could allow
guaranteed resources are overcommitted. (For example a parent queue with 100G
guaranteed memory, and there're 200 sub queues created under the parent, each
queue has 1G guaranteed memory, but most of the sub queues are not being used).
To solve the issue, scheduler may need to maintain a list of
{{#active-leaf-queues}} under one parent (An active-leaf-queue means a leaf
queue has at least one app not in final state). Parent queue's guaranteed
resource will be checked and enforced when state of leaf queue's changed to
active. Application submission will be rejected if
- {{Σ(leafQueue.guaranteed) (leafQueue ∈ \{active-leaf-queues\}) >
parent.guaranteed)}}.
> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue
> Mapping
> ----------------------------------------------------------------------------------
>
> Key: YARN-7117
> URL: https://issues.apache.org/jira/browse/YARN-7117
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacity scheduler
> Reporter: Wangda Tan
> Assignee: Wangda Tan
>
> Currently Capacity Scheduler doesn't support auto creation of queues when
> doing queue mapping. We saw more and more use cases which has complex queue
> mapping policies configured to handle application to queues mapping.
> The most common use case of CapacityScheduler queue mapping is to create one
> queue for each user/group. However update {{capacity-scheduler.xml}} and
> {{RMAdmin:refreshQueues}} needs to be done when new user/group onboard. One
> of the option to solve the problem is automatically create queues when new
> user/group arrives.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
