[
https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158854#comment-16158854
]
Clay B. commented on YARN-7117:
-------------------------------
As to my use-case:
I have an ad-hoc "users" queue which is a best-effort queue with some minimum
capacity granted to that overall queue. E.g. users can get their 4 vcpus and
4GB of RAM (or more if available), if the users best-effort queue is not
over-subscribed then they can pre-empt jobs running outside that queue for
their minimum.
However, on our Dev systems we may have 1,000+ developers and certainly we
don't expect them all to run at once (e.g. typical teleco-style
over-subscription assumptions) but we do want to ensure that other queues'
guaranteed capacity are not impacted and we want to ensure we have per-user
metrics on ad-hoc jobs which individual queues provide us via JMX over having a
single ad-hoc queue with YARN per-user metrics overall (where other queues may
be prudently used by a user and analysed).
Should a user submit when the parent queue is saturated it would be due to a
massive influx of users since, for me, one user has a minimum of 1% of the
overall queue capacity.
> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue
> Mapping
> ----------------------------------------------------------------------------------
>
> Key: YARN-7117
> URL: https://issues.apache.org/jira/browse/YARN-7117
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacity scheduler
> Reporter: Wangda Tan
> Assignee: Wangda Tan
>
> Currently Capacity Scheduler doesn't support auto creation of queues when
> doing queue mapping. We saw more and more use cases which has complex queue
> mapping policies configured to handle application to queues mapping.
> The most common use case of CapacityScheduler queue mapping is to create one
> queue for each user/group. However update {{capacity-scheduler.xml}} and
> {{RMAdmin:refreshQueues}} needs to be done when new user/group onboard. One
> of the option to solve the problem is automatically create queues when new
> user/group arrives.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
