Min Shen created YARN-5520:
------------------------------
Summary: [Capacity Scheduler] Change the logic for when to trigger
user/group mappings to queues
Key: YARN-5520
URL: https://issues.apache.org/jira/browse/YARN-5520
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.6.1, 2.7.0, 2.6.0
Reporter: Min Shen
In YARN-2411, the feature in Capacity Scheduler to support user/group based
mappings to queues was introduced.
In the original implementation, the configuration key
{{yarn.scheduler.capacity.queue-mappings-override.enable}} was added to control
when to enable overriding user requested queues.
However, even if this configuration is set to false, queue overriding could
still happen if the user didn't request for any specific queue or choose to
simply submit his job to "default" queue, according to the following if
condition which triggers queue overriding:
{code}
if (queueName.equals(YarnConfiguration.DEFAULT_QUEUE_NAME)
|| overrideWithQueueMappings)
{code}
This logic does not seem very reasonable, as there's no way to fully disable
queue overriding when mappings are configured inside capacity-scheduler.xml.
In addition, in our environment, we have setup a few organization dedicated
queues as well as some "adhoc" queues. The organization dedicated queues have
better resource guarantees and we want to be able to route users to the
corresponding organization queues. On the other hand, the "adhoc" queues have
less resource guarantees but everyone can use it to get some opportunistic
resources when the cluster is free.
The current logic will also prevent this type of use cases as when you enable
queue overriding, users cannot use these "adhoc" queues any more. They will
always be routed to the dedicated organization queues.
To address the above 2 issues, I propose to change the implementation so that:
* Admin can fully disable queue overriding even if mappings are already
configured.
* Admin have finer grained control to cope queue overriding with the above
mentioned organization/adhoc queue setups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
