[
https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966265#comment-13966265
]
Ashwin Shankar commented on YARN-1864:
--------------------------------------
[~octo47], thanks for your comments.
bq. 1. Placement rules coupled with queue definition. You need to check, that
queue was defined as hierarchical. It complicates a bit automatic config
generation.
No, we won't define any queue as 'hierarchical' with the new design and there
will be no checks. We just look at what the nested rule is returning and create
user queues underneath it. The queue returned by the nested rule can be any
parent queue in the hierarchy.
I think you are looking at my first patch(YARN1864-v1.patch),please ignore this
patch,we came up with the nested rule design after this patch was submitted.
bq. 2. You handle only one case, when user name should be added at the end. But
here can be more rules added, which will not require to create separate queues
for users
Not true. We can add other rules above and below hierarchicalUserQueue rule.
For eg:
<rule name="specified">
<rule name="hierarchicalUserQueue">
...//nested rule
</rule>
<rule name="default">
bq.In my sense, a bit cleaner solution should not expose couple queue
definitions and hierarchical nature of the rule (we ever don't need to
introduce such concept as 'hierarchical'). I purpose all logic should be placed
in base class and every rule can have two attributes: overridden parent and
overriden queue name..
We have a rule/concept called 'hierarchicalUserQueue' so that the 'userqueue'
under any parent functionality can be *decoupled* and can be used/not used with
any other rule. I understand we can use overridden 'parent','queuename' fields
but is I don't think it is an appropriate design considering rules like primary
group,secondary group doesn't need them.
bq. And additiona rules still needed, because right now you can't match users
by name or groups and place them elsewhere (not restricted by creating queue
named as group or secondary group)
Yes you are right,additional rules may be needed for matching user/groups with
a queue(parent/leaf),but this is orthogonal to this JIRA and can be implemented
independently. Note that even this rule can be nested under
hierarchicalUserQueue to get user queue support.
My intention is to put all queue placement decisions in one place,to make it
intuitive to configure for admin and make it extensible.
I think things will be a little more clear once I post a patch,please give me
couple of days.
> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>
> Key: YARN-1864
> URL: https://issues.apache.org/jira/browse/YARN-1864
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: scheduler
> Reporter: Ashwin Shankar
> Labels: scheduler
> Attachments: YARN-1864-v1.txt
>
>
> In Fair Scheduler, we want to be able to create user queues under any parent
> queue in the hierarchy. For eg. Say user1 submits a job to a parent queue
> called root.allUserQueues, we want be able to create a new queue called
> root.allUserQueues.user1 and run user1's job in it.Any further jobs submitted
> by this user to root.allUserQueues will be run in this newly created
> root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which
> creates user queues under root queue. But we want the ability to create user
> queues under ANY parent queue.
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other
> if its fair share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair
> share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume
> only a fraction of resources in the shared cluster. By creating this
> feature,we could do that by giving a fair share to the parent user queue
> which is then redistributed to all the dynamically created user queues.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
