[
https://issues.apache.org/jira/browse/YARN-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Stepachev updated YARN-1921:
---
Description:
Fair scheduler has a couple of QueuePlacementRules. Those rules can create
queues, if they not exists with hardcoded prefix root..
Consider an example: we have a placement rule, which creates user's queue if it
not exists. Current implementation creates it at root. prefix Suppose that
this user runs a big job. In that case it will get a fair share of resources
because queue will be created at 'root.' with default settings, and that
affects all other users of the cluster.
Of course, FairScheduler can place such users to default queue, but in that
case if user submits a big queue it will eat resources of whole queue, and we
know that no preemption can be done within one queue (Or i'm wrong?). So
effectively one user can usurp all default queue resources.
To solve that I created a patch, which allows to override root. prefix in
QueuePlacementRules. Thats gives us flexibility to automatically create queues
for users or group of users under predefined queue. So, every user will get a
separate queue and will share parent queue resources and can't usurp all
resources, because parent node can be configured to preempt tasks.
Consider example (parent queue specified for each rule):
{code:title=policy.xml|borderStyle=solid}
queuePlacementPolicy
rule name='specified' parent='granted'/
rule name='user' parent='guests'/
/queuePlacementPolicy
{code}
With such definition queue requirements will give us:
{code:title=Example.java|borderStyle=solid}
root.granted.specifiedq == policy.assignAppToQueue(specifiedq, someuser);
root.guests.someuser == policy.assignAppToQueue(default, someuser);
root.guests.otheruser == policy.assignAppToQueue(default, otheruser);
{code}
was:
Fair scheduler has a couple of QueuePlacementRules. Those rules can create
queues, if they not exists with hardcoded prefix root..
Consider an example: we have a placement rule, which creates user's queue if it
not exists. Current implementation creates it at root. prefix Suppose that
this user runs a big job. In that case it will get a fair share of resources
because queue will be created at 'root.' with default settings, and that
affects all other users of the cluster.
Of course, FairScheduler can place such users to default queue, but in that
case if user submits a big queue it will eats resources of whole queue, and we
know that no preemption can be done within one queue (Or i'm wrong?). So
effectively one user can usurp all default queue resources.
To solve that I created a patch, which allows to override root. prefix in
QueuePlacementRules. Thats gives us flexibility to automatically create queues
for users or group of users under predefined queue. So, every user will get a
separate queue and will share parent queue resources and can't usurp all
resources, because parent node can be configured to preempt tasks.
Consider example (parent queue specified for each rule):
{code:title=policy.xml|borderStyle=solid}
queuePlacementPolicy
rule name='specified' parent='granted'/
rule name='user' parent='guests'/
/queuePlacementPolicy
{code}
With such definition queue requirements will give us:
{code:title=Example.java|borderStyle=solid}
root.granted.specifiedq == policy.assignAppToQueue(specifiedq, someuser);
root.guests.someuser == policy.assignAppToQueue(default, someuser);
root.guests.otheruser == policy.assignAppToQueue(default, otheruser);
{code}
Allow to override queue prefix, where new queues created
Key: YARN-1921
URL: https://issues.apache.org/jira/browse/YARN-1921
Project: Hadoop YARN
Issue Type: Improvement
Components: scheduler
Affects Versions: 2.3.0
Environment: Yarn 2.3.0
Reporter: Andrey Stepachev
Attachments: YARN-1921.patch
Fair scheduler has a couple of QueuePlacementRules. Those rules can create
queues, if they not exists with hardcoded prefix root..
Consider an example: we have a placement rule, which creates user's queue if
it not exists. Current implementation creates it at root. prefix Suppose
that this user runs a big job. In that case it will get a fair share of
resources because queue will be created at 'root.' with default settings, and
that affects all other users of the cluster.
Of course, FairScheduler can place such users to default queue, but in that
case if user submits a big queue it will eat resources of whole queue, and we
know that no preemption can be done within one queue (Or i'm wrong?). So
effectively one user can usurp all default queue resources.
To solve that I created a patch, which allows to override root. prefix in
QueuePlacementRules. Thats gives us flexibility to
