[
https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966222#comment-13966222
]
Andrey Stepachev commented on YARN-1864:
----------------------------------------
Ashwin, I understand your solution. It is good and generic.
But in my sense it a bit complicated.
1. Placement rules coupled with queue definition. You need to check, that queue
was defined as hierarchical. It complicates a bit automatic config generation.
2. You handle only one case, when user name should be added at the end. But
here can be more rules added, which will not require to create separate queues
for users.
In my sense, a bit cleaner solution should not expose couple queue definitions
and hierarchical nature of the rule (we ever don't need to introduce such
concept as 'hierarchical'). I purpose all logic should be placed in base class
and every rule can have two attributes: overridden parent and overriden queue
name (my patch makes each rule decides separately, but it can be easily done by
protected method makeFinalQueueName(suggestedQueueName) in each rule right
before return queue name).
Moreover, we can add tag 'parent' and automatically apply attributes to inner
rules. I.e.
{code}
<parent queueName="some.parent" appendUser="true">
<rule ... />
<rule ... />
</parent>
{code}
What can be achieved:
Don't need complicated logic with keeping list of hierarchical queus and match
them.
Each rule defines queue names locally in placement rule definitions.
> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>
> Key: YARN-1864
> URL: https://issues.apache.org/jira/browse/YARN-1864
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: scheduler
> Reporter: Ashwin Shankar
> Labels: scheduler
> Attachments: YARN-1864-v1.txt
>
>
> In Fair Scheduler, we want to be able to create user queues under any parent
> queue in the hierarchy. For eg. Say user1 submits a job to a parent queue
> called root.allUserQueues, we want be able to create a new queue called
> root.allUserQueues.user1 and run user1's job in it.Any further jobs submitted
> by this user to root.allUserQueues will be run in this newly created
> root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which
> creates user queues under root queue. But we want the ability to create user
> queues under ANY parent queue.
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other
> if its fair share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair
> share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume
> only a fraction of resources in the shared cluster. By creating this
> feature,we could do that by giving a fair share to the parent user queue
> which is then redistributed to all the dynamically created user queues.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
