[
https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980569#comment-13980569
]
Ashwin Shankar commented on YARN-1864:
--------------------------------------
[~sandyr],
bq. Can we use configuredQueues for this? We shouldn't be touching the
QueueManager from inside the QueuePlacementPolicy.
Yes we can use configuredQueues for this(instead of queueMgr code) like the
following,but there is a case we will miss :
{code}
if (configuredQueues.get(FSQueueType.LEAF).contains(nestedRuleOutput)) {
return "";
}
{code}
Here is the problem with above code. There are five types of queues the nested
rule can return :
1. Existing Configured Parent Queue.
2. Existing Configured Leaf Queue.
3. Existing Unconfigured Parent Queue(created when create=true)
4. Existing Unconfigured Leaf Queue(created when create=true)
5. Non existent queue.
Using above code snippet handles all the cases correctly except case4.
When we get an existing unconfigured leaf queue from nested rule,we will not
skip the rule
and would result in an error later.
Using queueMgr would cover all 5 cases.
Having said that,case-4 occurs in corner cases,like when the admin makes a
regular rule into a nested rule
without bouncing RM. For eg : Initially he might have had primaryGroup with
create=true as a regular rule.
This could create some unconfigured leaf queues. Then he nests this
primaryGroup under userQueueBelow rule.
Now those unconfigured leaf queues can again be returned by the nested rule,
and will not be caught by the code snippet
mentioned above and user would get an error later. But using queueMgr would
cause a skip and hopefully would run the app in
some other queue.
Let me know if you want to support this ? If not, configuresQueues will
work,I'll change that in the next patch.
Also were there any other comments you had in mind which you've not mentioned
yet ?
> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>
> Key: YARN-1864
> URL: https://issues.apache.org/jira/browse/YARN-1864
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: scheduler
> Reporter: Ashwin Shankar
> Labels: scheduler
> Attachments: YARN-1864-v1.txt, YARN-1864-v2.txt, YARN-1864-v3.txt
>
>
> In Fair Scheduler, we want to be able to create user queues under any parent
> queue in the hierarchy. For eg. Say user1 submits a job to a parent queue
> called root.allUserQueues, we want be able to create a new queue called
> root.allUserQueues.user1 and run user1's job in it.Any further jobs submitted
> by this user to root.allUserQueues will be run in this newly created
> root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which
> creates user queues under root queue. But we want the ability to create user
> queues under ANY parent queue.
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other
> if its fair share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair
> share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume
> only a fraction of resources in the shared cluster. By creating this
> feature,we could do that by giving a fair share to the parent user queue
> which is then redistributed to all the dynamically created user queues.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
