[
https://issues.apache.org/jira/browse/YARN-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429099#comment-15429099
]
Wangda Tan commented on YARN-5520:
----------------------------------
Thanks for sharing your thoughts, [~mshen], [~Ying Zhang], [~venkateshrin].
Since we have a pluggable mechanism for queue placement policy since YARN-3635,
is it fine for you to add a new placement policy to supporting new queue policy
mapping requirements? I don't want to make changes unless such new requirements
are generic for many users. This is because different users have quite
different requirements for queue placement policies.
Now we don't support loading queue placement rules from configuration, it's
necessary to have some minor changes to enable it, if anyone can open a JIRA
and working on the patch, I can help with patch review.
> [Capacity Scheduler] Change the logic for when to trigger user/group mappings
> to queues
> ---------------------------------------------------------------------------------------
>
> Key: YARN-5520
> URL: https://issues.apache.org/jira/browse/YARN-5520
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 2.6.0, 2.7.0, 2.6.1
> Reporter: Min Shen
>
> In YARN-2411, the feature in Capacity Scheduler to support user/group based
> mappings to queues was introduced.
> In the original implementation, the configuration key
> {{yarn.scheduler.capacity.queue-mappings-override.enable}} was added to
> control when to enable overriding user requested queues.
> However, even if this configuration is set to false, queue overriding could
> still happen if the user didn't request for any specific queue or choose to
> simply submit his job to "default" queue, according to the following if
> condition which triggers queue overriding:
> {code}
> if (queueName.equals(YarnConfiguration.DEFAULT_QUEUE_NAME)
> || overrideWithQueueMappings)
> {code}
> This logic does not seem very reasonable, as there's no way to fully disable
> queue overriding when mappings are configured inside capacity-scheduler.xml.
> In addition, in our environment, we have setup a few organization dedicated
> queues as well as some "adhoc" queues. The organization dedicated queues have
> better resource guarantees and we want to be able to route users to the
> corresponding organization queues. On the other hand, the "adhoc" queues have
> less resource guarantees but everyone can use it to get some opportunistic
> resources when the cluster is free.
> The current logic will also prevent this type of use cases as when you enable
> queue overriding, users cannot use these "adhoc" queues any more. They will
> always be routed to the dedicated organization queues.
> To address the above 2 issues, I propose to change the implementation so that:
> * Admin can fully disable queue overriding even if mappings are already
> configured.
> * Admin have finer grained control to cope queue overriding with the above
> mentioned organization/adhoc queue setups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
