[
https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145335#comment-16145335
]
Jason Lowe commented on YARN-7117:
----------------------------------
The main issue I see is that it could end up creating what previously was an
invalid queue setup where the sum of child capacities != 100%. For example, if
each user gets a queue, each queue is "guaranteed" 10GB, and the parent queue
only has 100GB. It's weird when 100 users show up because then we have child
queues being configured with capacities far beyond the parent queue's
capabilities which is something CapacityScheduler never allowed before. If
auto-queues can do this, do we also allow it for manually configured ones?
It works well when only 10 users are ever active at once, but what happens when
50 are? What does the admin tell them about their "guarantees" at that point?
IMHO if we're going to do this then either there needs to be guarantees and
attempts to auto-create a queue beyond the parent's capability fails, or queues
are created with no guarantees (capacity = 0%, max capacity = 100%) because
that's essentially what we're giving them -- no guarantees but best-effort
capacity based on what others are doing.
In short, if we're going to allow new, previously illegal configurations for
auto-queues then arguably we should be consistent and allow them elsewhere as
well.
> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue
> Mapping
> ----------------------------------------------------------------------------------
>
> Key: YARN-7117
> URL: https://issues.apache.org/jira/browse/YARN-7117
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacity scheduler
> Reporter: Wangda Tan
> Assignee: Wangda Tan
>
> Currently Capacity Scheduler doesn't support auto creation of queues when
> doing queue mapping. We saw more and more use cases which has complex queue
> mapping policies configured to handle application to queues mapping.
> The most common use case of CapacityScheduler queue mapping is to create one
> queue for each user/group. However update {{capacity-scheduler.xml}} and
> {{RMAdmin:refreshQueues}} needs to be done when new user/group onboard. One
> of the option to solve the problem is automatically create queues when new
> user/group arrives.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
