[ https://issues.apache.org/jira/browse/YARN-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xianyin Xin updated YARN-4134: ------------------------------ Description: Now FairScheudler uses a choose-a-candidate method to select a container from leaf queues that to be preempted, in {{FSParentQueue.preemptContainer()}}, {code} readLock.lock(); try { for (FSQueue queue : childQueues) { if (candidateQueue == null || comparator.compare(queue, candidateQueue) > 0) { candidateQueue = queue; } } } finally { readLock.unlock(); } // Let the selected queue choose which of its container to preempt if (candidateQueue != null) { toBePreempted = candidateQueue.preemptContainer(); } {code} a candidate child queue is selected. However, if the queue's usage isn't over it's fairshare, preemption will not happen: {code} if (!preemptContainerPreCheck()) { return toBePreempted; } {code} A scenario: {code} root / \ queue1 queue2 / \ queue2.3, ( queue2.4 ) {code} suppose there're 8 containers, and queues at any level have the same weight. queue1 takes 4 and queue2.3 takes 4, so both queue1 and queue2 are at their fairshare. Now we submit an app in queue2.4 with 4 containers needs, it should preempt 2 from queue2.3, but the candidate-containers selection procedure will stop at queue1, so none of the containers will be preempted. was: Now FairScheudler uses a choose-a-candidate method to select a container from leaf queues that to be preempted, in {{FSParentQueue.preemptContainer()}}, {code} readLock.lock(); try { for (FSQueue queue : childQueues) { if (candidateQueue == null || comparator.compare(queue, candidateQueue) > 0) { candidateQueue = queue; } } } finally { readLock.unlock(); } // Let the selected queue choose which of its container to preempt if (candidateQueue != null) { toBePreempted = candidateQueue.preemptContainer(); } {code} a candidate child queue is selected. However, if the queue's usage isn't over it's fairshare, preemption will not happen: {code} if (!preemptContainerPreCheck()) { return toBePreempted; } {code} A scenario: {code} root / \ queue1 queue2 / \ queue1.3, ( queue1.4 ) {code} suppose there're 8 containers, and queues at any level have the same weight. queue1.3 takes 4 and queue2 takes 4, so both queue1 and queue2 are at their fairshare. Now we submit an app in queue1.4 with 4 containers needs, it should preempt 2 from queue1.3, but the candidate-containers selection procedure will stop at level that all of the child queues are not over their fairshare, and none of the containers will be preempted. > FairScheduler preemption stops at queue level that all child queues are not > over their fairshare > ------------------------------------------------------------------------------------------------ > > Key: YARN-4134 > URL: https://issues.apache.org/jira/browse/YARN-4134 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler > Reporter: Xianyin Xin > > Now FairScheudler uses a choose-a-candidate method to select a container from > leaf queues that to be preempted, in {{FSParentQueue.preemptContainer()}}, > {code} > readLock.lock(); > try { > for (FSQueue queue : childQueues) { > if (candidateQueue == null || > comparator.compare(queue, candidateQueue) > 0) { > candidateQueue = queue; > } > } > } finally { > readLock.unlock(); > } > // Let the selected queue choose which of its container to preempt > if (candidateQueue != null) { > toBePreempted = candidateQueue.preemptContainer(); > } > {code} > a candidate child queue is selected. However, if the queue's usage isn't over > it's fairshare, preemption will not happen: > {code} > if (!preemptContainerPreCheck()) { > return toBePreempted; > } > {code} > A scenario: > {code} > root > / \ > queue1 queue2 > / \ > queue2.3, ( queue2.4 ) > {code} > suppose there're 8 containers, and queues at any level have the same weight. > queue1 takes 4 and queue2.3 takes 4, so both queue1 and queue2 are at their > fairshare. Now we submit an app in queue2.4 with 4 containers needs, it > should preempt 2 from queue2.3, but the candidate-containers selection > procedure will stop at queue1, so none of the containers will be preempted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)