[
https://issues.apache.org/jira/browse/YARN-9879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009975#comment-17009975
]
Wangda Tan commented on YARN-9879:
----------------------------------
[~pbacsko], thanks for working on the design.
In general, I agree with what [~wilfreds] mentioned: we should try to avoid
change RPC protocols, instead we just change internal logic to make sure
multiple queues can be handled.
To me there're two major parts:
1) Whatever logic inside CS to allow multiple queue names. Either solution
mentioned in the comment:
https://issues.apache.org/jira/browse/YARN-9879?focusedCommentId=17009845&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17009845
should be fine. And I expect the lookup of queue name (not queue path) should
only be called when submit application.
And once application is submitted to CS, internal to CS, we should make sure we
use queue path instead of queue name at all other places. Otherwise we will
complicate other logics.
2) When submit app, the scheduler going to accept/reject app based on the
uniqueness of queue name or path specified. The core part need to be changed is
inside RMAppManager:
{code:java}
if (!isRecovery && YarnConfiguration.isAclEnabled(conf)) {
if (scheduler instanceof CapacityScheduler) {
String queueName = submissionContext.getQueue();
String appName = submissionContext.getApplicationName();
CSQueue csqueue = ((CapacityScheduler) scheduler).getQueue(queueName);{code}
Instead of using scheduler.getQueue, we may need to consider to add a method
like getAppSubmissionQueue() to get a queue based on path or name, and after
that, we will put normalized queue_path back to submission context of
application to make sure in the future inside scheduler we all refer to queue
path.
For the comment from [~wilfreds]:
{quote}The important part is applying a new configuration. If the configuration
adds a leaf queue that is not unique the configuration update currently is
rejected. With this change we would allow that config to become active. This
*could* break existing applications when they try to submit to the leaf queue
that is no longer unique.
{quote}
I personally think it is not a big deal if application reject reasons from RM
can clearly guide users to use full qualified queue path when duplicated queue
names exists. It is like if a team has only one Peter we can use the first name
only otherwise we will add last name to avoid confusion. It isn't
counter-intuitive to me.
Also, we need to handle queue mapping for queue-path instead of queue name
also, I didn't see it from the design doc or I missed it.
> Allow multiple leaf queues with the same name in CS
> ---------------------------------------------------
>
> Key: YARN-9879
> URL: https://issues.apache.org/jira/browse/YARN-9879
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Gergely Pollak
> Assignee: Gergely Pollak
> Priority: Major
> Attachments: DesignDoc_v1.pdf
>
>
> Currently the leaf queue's name must be unique regardless of its position in
> the queue hierarchy.
> Design doc and first proposal is being made, I'll attach it as soon as it's
> done.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
