Ashwin Shankar created YARN-1961:
------------------------------------
Summary: Fair scheduler preemption doesn't work for non-leaf queues
Key: YARN-1961
URL: https://issues.apache.org/jira/browse/YARN-1961
Project: Hadoop YARN
Issue Type: Bug
Components: scheduler
Affects Versions: 2.4.0
Reporter: Ashwin Shankar
Setting minResources and minSharePreemptionTimeout to a non-leaf queue doesn't
cause preemption to happen when that non-leaf queue is below minResources and
there are outstanding demands in that non-leaf queue.
Here is an example fs allocation config(partial) :
{code:xml}
<queue name="abc">
<minResources>3072 mb,0 vcores</minResources>
<minSharePreemptionTimeout>30</minSharePreemptionTimeout>
<queue name="childabc1">
</queue>
<queue name="childabc2">
</queue>
</queue>
{code}
With the above configs,preemption doesn't seem to happen if queue abc is below
minShare and it has outstanding unsatisfied demands from apps in its child
queues. Ideally in such cases we would like preemption to kick off and reclaim
resources from other queues(not under queue abc).
Looking at the code it seems like preemption checks for starvation only at the
leaf queue level and not at the parent level.
{code:title=FairScheduler.java|borderStyle=solid}
boolean isStarvedForMinShare(FSLeafQueue sched)
boolean isStarvedForFairShare(FSLeafQueue sched)
{code}
This affects our use case where we have a parent queue with probably a 100
unconfigured leaf queues under it.We want to give a minshare to the parent
queue to protect all the leaf queues under it,but we cannot do it due to this
bug.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
