[
https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated YARN-1864:
---------------------------------
Attachment: YARN-1864-v3.txt
[~sandyr] Hey Sandy,Patch refreshed with most of your comments.
bq. Any reason why it can't?
I removed this code in the latest patch.
I put that previously since it didn't make sense to me why one would want to
create 'user queues under a user queue'. But I'm fine not having this
restriction.
bq. The QueuePlacementPolicy isn't in responsible for verifying this. We expect
to hit an error later on if the queue returned by the QueuePlacementPolicy
isn't a leaf queue. This can happen with other rules, for example with the
"specified" rule if someone tries to explicitly submit to a parent queue.
Yes makes sense,but here is a use case which I'm trying to support where user
doesn't specify a queue.
*Use case* : Nested rule is also a regular rule
{code:xml}
<queuePlacementPolicy>
<rule name="userQueueBelow">
<rule name="primaryGroup" create="false">
</rule>
<rule name="primaryGroup" create="false">
{code}
If we want some primary group queues to have user queues underneath and some
primary groups to be plain leaf queues,we could
create a queue placement policy like above. In the 'userQueueBelow' rule,if we
don't check if the nested rule returned a leaf queue
and skip,we will never reach the 3rd rule and always throw an error later when
user's primary group is a leaf queue.
bq. Along similar lines to the previous comment, I don't think we need this
logic, or isNestedQueue and FSQueueType, in the queue placement policy. If the
rule places an app in "root.engineering.ashwin" and root.engineering happens to
be a leaf or root.ashwin happens to be a parent, we'll end up throwing an error
later, which is about the best we can do. Let me know if there are cases I'm
missing or if I'm not thinking this through deeply enough.
Removed isNested logic in latest patch. But FSQueueType is needed to get
configured parent queues and configured leaf queues.
Currently in hadoop trunk code, 'configuredQueues' in QueuePlacementRule.java
is a Set containing only configured leaf queues.
However with userQueueBelow rule,the nested rule can return configured parent
queues when create is false.
bq. This has gotten to be a fairly big patch - mind making these changed to the
"default" rule in a separate JIRA?
Sure,I've removed changes to default rule in the patch,I'll create a new jira
for this.
bq.The name "HierarchicalUserQueue" seems a kind of weird to me.
Yes I agree. I think your suggestion 'UserQueueBelow' seems reasonable.I've
changed the name at all places in the latest patch.
bq. I've got a few stylistic comments in addition to these.
Sure please let me know,I'll incorporate it in my next patch.
> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>
> Key: YARN-1864
> URL: https://issues.apache.org/jira/browse/YARN-1864
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: scheduler
> Reporter: Ashwin Shankar
> Labels: scheduler
> Attachments: YARN-1864-v1.txt, YARN-1864-v2.txt, YARN-1864-v3.txt
>
>
> In Fair Scheduler, we want to be able to create user queues under any parent
> queue in the hierarchy. For eg. Say user1 submits a job to a parent queue
> called root.allUserQueues, we want be able to create a new queue called
> root.allUserQueues.user1 and run user1's job in it.Any further jobs submitted
> by this user to root.allUserQueues will be run in this newly created
> root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which
> creates user queues under root queue. But we want the ability to create user
> queues under ANY parent queue.
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other
> if its fair share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair
> share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume
> only a fraction of resources in the shared cluster. By creating this
> feature,we could do that by giving a fair share to the parent user queue
> which is then redistributed to all the dynamically created user queues.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
