[
https://issues.apache.org/jira/browse/YARN-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419734#comment-15419734
]
Min Shen commented on YARN-5520:
--------------------------------
Will attach patch soon.
> [Capacity Scheduler] Change the logic for when to trigger user/group mappings
> to queues
> ---------------------------------------------------------------------------------------
>
> Key: YARN-5520
> URL: https://issues.apache.org/jira/browse/YARN-5520
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 2.6.0, 2.7.0, 2.6.1
> Reporter: Min Shen
>
> In YARN-2411, the feature in Capacity Scheduler to support user/group based
> mappings to queues was introduced.
> In the original implementation, the configuration key
> {{yarn.scheduler.capacity.queue-mappings-override.enable}} was added to
> control when to enable overriding user requested queues.
> However, even if this configuration is set to false, queue overriding could
> still happen if the user didn't request for any specific queue or choose to
> simply submit his job to "default" queue, according to the following if
> condition which triggers queue overriding:
> {code}
> if (queueName.equals(YarnConfiguration.DEFAULT_QUEUE_NAME)
> || overrideWithQueueMappings)
> {code}
> This logic does not seem very reasonable, as there's no way to fully disable
> queue overriding when mappings are configured inside capacity-scheduler.xml.
> In addition, in our environment, we have setup a few organization dedicated
> queues as well as some "adhoc" queues. The organization dedicated queues have
> better resource guarantees and we want to be able to route users to the
> corresponding organization queues. On the other hand, the "adhoc" queues have
> less resource guarantees but everyone can use it to get some opportunistic
> resources when the cluster is free.
> The current logic will also prevent this type of use cases as when you enable
> queue overriding, users cannot use these "adhoc" queues any more. They will
> always be routed to the dedicated organization queues.
> To address the above 2 issues, I propose to change the implementation so that:
> * Admin can fully disable queue overriding even if mappings are already
> configured.
> * Admin have finer grained control to cope queue overriding with the above
> mentioned organization/adhoc queue setups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
