[
https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948349#comment-13948349
]
Ashwin Shankar commented on YARN-1864:
--------------------------------------
Thanks Sandy for looking into this !
bq. I'd like to see this work with queue placement policies in the sense that
one should not always need to explicitly specify the parent queue when using
this feature. The queue placement policy should be able to handle both choosing
the parent queue and the user queue underneath.
I did think of making this work without users needing to specify a parent but
how can we associate a user to a parentUserQueue(assuming multiple
parentUserQueues)? Like primary and secondary queue placement policies had
users being associated with prim/sec groups.
If we are assuming allowing only one parentUserQueue in the cluster then yes,
we can associate ALL users with that parent. But are we thinking of restricting
to only one parentUserQueue ?
> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>
> Key: YARN-1864
> URL: https://issues.apache.org/jira/browse/YARN-1864
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: scheduler
> Reporter: Ashwin Shankar
> Labels: scheduler
> Attachments: YARN-1864-v1.txt
>
>
> In Fair Scheduler, we want to be able to create user queues under any parent
> queue in the hierarchy. For eg. Say user1 submits a job to a parent queue
> called root.allUserQueues, we want be able to create a new queue called
> root.allUserQueues.user1 and run user1's job in it.Any further jobs submitted
> by this user to root.allUserQueues will be run in this newly created
> root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which
> creates user queues under root queue. But we want the ability to create user
> queues under ANY parent queue.
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other
> if its fair share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair
> share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume
> only a fraction of resources in the shared cluster. By creating this
> feature,we could do that by giving a fair share to the parent user queue
> which is then redistributed to all the dynamically created user queues.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
