[
https://issues.apache.org/jira/browse/YARN-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186273#comment-15186273
]
Karthik Kambatla commented on YARN-1961:
----------------------------------------
Configuring minResources and minshare preemption on non-leaf queues might not
be desirable for the following reasons:
# Minshare preemption was often used in the past to address cases where users
wanted to be able to trigger preemption when a queue is over half of its
fairshare. Now that we have added fairsharePreemptionThreshold, this is less
the case.
# Other usecase for minshare preemption was to give enough resources for a high
priority job quickly (shorter minshare preemption timeout). Now, we support
setting the fairsharePreemptionTimeout per queue. Also, while it might be
easier to figure out resources need for an application, it is much harder to
come up with absolute minimum resources required for any queue in general and a
non-leaf queue in particular.
[~ashwinshankar77] - if you agree this is less desirable, I would like to close
this as "Won't Fix".
> Fair scheduler preemption doesn't work for non-leaf queues
> ----------------------------------------------------------
>
> Key: YARN-1961
> URL: https://issues.apache.org/jira/browse/YARN-1961
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, scheduler
> Affects Versions: 2.4.0
> Reporter: Ashwin Shankar
> Assignee: Ashwin Shankar
> Labels: scheduler
>
> Setting minResources and minSharePreemptionTimeout to a non-leaf queue
> doesn't cause preemption to happen when that non-leaf queue is below
> minResources and there are outstanding demands in that non-leaf queue.
> Here is an example fs allocation config(partial) :
> {code:xml}
> <queue name="abc">
> <minResources>3072 mb,0 vcores</minResources>
> <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
> <queue name="childabc1">
> </queue>
> <queue name="childabc2">
> </queue>
> </queue>
> {code}
> With the above configs,preemption doesn't seem to happen if queue abc is
> below minShare and it has outstanding unsatisfied demands from apps in its
> child queues. Ideally in such cases we would like preemption to kick off and
> reclaim resources from other queues(not under queue abc).
> Looking at the code it seems like preemption checks for starvation only at
> the leaf queue level and not at the parent level.
> {code:title=FairScheduler.java|borderStyle=solid}
> boolean isStarvedForMinShare(FSLeafQueue sched)
> boolean isStarvedForFairShare(FSLeafQueue sched)
> {code}
> This affects our use case where we have a parent queue with probably a 100
> unconfigured leaf queues under it.We want to give a minshare to the parent
> queue to protect all the leaf queues under it,but we cannot do it due to this
> bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
