[
https://issues.apache.org/jira/browse/YARN-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115847#comment-16115847
]
Steven Rand edited comment on YARN-6956 at 8/6/17 4:35 PM:
-----------------------------------------------------------
Hi [[email protected]], thanks for the quick reply and explanation. That
concern definitely makes sense, and in general YARN-6163 seems like a good
change.
However, what I'm seeing is that only considering RRs for one node actually
causes some of my apps to remain starved for quite a long time. The series of
events that happens in a loop is:
1. The app is correctly considered to be starved
2. The app has many RRs, several of which can be satisfied, but only one RR is
actually considered for preemption as per this JIRA's description
3. That particular RR happens to be for a node on which no containers can be
preempted for the app, so the app remains starved
Since the order of the list of RRs is the same each time through the loop, the
same RR is always considered, no containers are preempted, and the app remains
starved, even though it has other RRs that could be satisfied.
I haven't thought enough yet about what a solution would look like, but it
seems like we should be able to keep the benefits of YARN-6163 while also
avoiding this issue. I'll try to have a patch within the next few days if
people agree that we should change the behavior.
was (Author: steven rand):
Hi [[email protected]], thanks for the quick reply and explanation. That
concern definitely makes sense, and in general YARN-6163 seems like a good
change.
However, what I'm seeing is that only considering RRs for one node actually
causes some of my apps to remain starved for quite a long time. The series of
events that happens in a loop is:
1. The app is correctly considered to be starved
2. The app has many RRs, several of which can be satisfied, but only one RR is
actually considered for preemption as per this JIRA's description
3. That particular RR happens to be for a node on which the no containers can
be preempted for the app, so the app remains starved
Since the order of the list of RRs is the same each time through the loop, the
same RR is always considered, no containers are preempted, and the app remains
starved, even though it has other RRs that could be satisfied.
I haven't thought enough yet about what a solution would look like, but it
seems like we should be able to keep the benefits of YARN-6163 while also
avoiding this issue. I'll try to have a patch within the next few days if
people agree that we should change the behavior.
> preemption may only consider resource requests for one node
> -----------------------------------------------------------
>
> Key: YARN-6956
> URL: https://issues.apache.org/jira/browse/YARN-6956
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
> Affects Versions: 2.9.0, 3.0.0-beta1
> Environment: CDH 5.11.0
> Reporter: Steven Rand
>
> I'm observing the following series of events on a CDH 5.11.0 cluster, which
> seem to be possible after YARN-6163:
> 1. An application is considered to be starved, so {{FSPreemptionThread}}
> calls {{identifyContainersToPreempt}}, and that calls
> {{FSAppAttempt#getStarvedResourceRequests}} to get a list of
> {{ResourceRequest}} instances that are enough to address the app's starvation.
> 2. The first {{ResourceRequest}} that {{getStarvedResourceRequests}} sees is
> enough to address the app's starvation, so we break out of the loop over
> {{appSchedulingInfo.getAllResourceRequests()}} after only one iteration:
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L1180.
> We return only this one {{ResourceRequest}} back to the
> {{identifyContainersToPreempt}} method.
> 3. It turns out that this particular {{ResourceRequest}} happens to have a
> value for {{getResourceName}} that identifies a specific node in the cluster.
> This causes preemption to only consider containers on that node, and not the
> rest of the cluster.
> [~kasha], does that make sense? I'm happy to submit a patch if I'm
> understanding the problem correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
