[ 
https://issues.apache.org/jira/browse/YARN-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115847#comment-16115847
 ] 

Steven Rand commented on YARN-6956:
-----------------------------------

Hi [~dan...@cloudera.com], thanks for the quick reply and explanation. That 
concern definitely makes sense, and in general YARN-6163 seems like a good 
change.

However, what I'm seeing is that only considering RRs for one node actually 
causes some of my apps to remain starved for quite a long time. The series of 
events that happens in a loop is:

1. The app is correctly considered to be starved
2. The app has many RRs, several of which can be satisfied, but only one RR is 
actually considered for preemption as per this JIRA's description
3. That particular RR happens to be for a node on which the no containers can 
be preempted for the app, so the app remains starved

Since the order of the list of RRs is the same each time through the loop, the 
same RR is always considered, no containers are preempted, and the app remains 
starved, even though it has other RRs that could be satisfied.

I haven't thought enough yet about what a solution would look like, but it 
seems like we should be able to keep the benefits of YARN-6163 while also 
avoiding this issue. I'll try to have a patch within the next few days if 
people agree that we should change the behavior.

> preemption may only consider resource requests for one node
> -----------------------------------------------------------
>
>                 Key: YARN-6956
>                 URL: https://issues.apache.org/jira/browse/YARN-6956
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.9.0, 3.0.0-beta1
>         Environment: CDH 5.11.0
>            Reporter: Steven Rand
>
> I'm observing the following series of events on a CDH 5.11.0 cluster, which 
> seem to be possible after YARN-6163:
> 1. An application is considered to be starved, so {{FSPreemptionThread}} 
> calls {{identifyContainersToPreempt}}, and that calls 
> {{FSAppAttempt#getStarvedResourceRequests}} to get a list of 
> {{ResourceRequest}} instances that are enough to address the app's starvation.
> 2. The first {{ResourceRequest}} that {{getStarvedResourceRequests}} sees is 
> enough to address the app's starvation, so we break out of the loop over 
> {{appSchedulingInfo.getAllResourceRequests()}} after only one iteration: 
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L1180.
>  We return only this one {{ResourceRequest}} back to the 
> {{identifyContainersToPreempt}} method.
> 3. It turns out that this particular {{ResourceRequest}} happens to have a 
> value for {{getResourceName}} that identifies a specific node in the cluster. 
> This causes preemption to only consider containers on that node, and not the 
> rest of the cluster.
> [~kasha], does that make sense? I'm happy to submit a patch if I'm 
> understanding the problem correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to