[
https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145976#comment-16145976
]
Jason Lowe commented on YARN-7117:
----------------------------------
bq. Adding queue with zero guaranteed resource is one possible solution, but it
may not be good enough for resource sharing since many users don't frequently
use cluster.
If we get the user limit and user limit factor settings correct for the queue
(i.e.: make sure the user limit is 100% and ULF is large enough to completely
fill the parent queue) then I think it will work as intended without needing to
worry about how many other, unused queues there are. The scheduler will
automatically try to balance the load between all the active queues just like
it does with the regular queues today. And with the ULF being large enough,
the other queues should have no trouble stealing from any underutilized sibling
queues.
bq. Instead, maybe admin can configure a "delete-policy" which delete queues
which are unused for X secs. Thoughts?
If we have auto-create then I guess it could make sense to have auto-delete,
although it may be confusing to users that were querying their auto-created
queue status when suddenly that status request fails due to a non-existent
queue error. Ripping it out may be tricky depending upon the expectations of
the user.
If we choose to provide some non-zero capacity (separate from max-capacity) for
auto-created queues, What happens when more users become active than we can
provide guarantees for in the parent queue? Does the job submission fail since
it cannot create the child queue with that guarantee or ..?
I don't have all the details on the specific use cases, but this seems like
we're going out of our way to essentially emulate what user limits and in-queue
preemption can already accomplish when users share the same queue.
> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue
> Mapping
> ----------------------------------------------------------------------------------
>
> Key: YARN-7117
> URL: https://issues.apache.org/jira/browse/YARN-7117
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacity scheduler
> Reporter: Wangda Tan
> Assignee: Wangda Tan
>
> Currently Capacity Scheduler doesn't support auto creation of queues when
> doing queue mapping. We saw more and more use cases which has complex queue
> mapping policies configured to handle application to queues mapping.
> The most common use case of CapacityScheduler queue mapping is to create one
> queue for each user/group. However update {{capacity-scheduler.xml}} and
> {{RMAdmin:refreshQueues}} needs to be done when new user/group onboard. One
> of the option to solve the problem is automatically create queues when new
> user/group arrives.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
