[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352866#comment-16352866
]
Yufei Gu edited comment on YARN-7655 at 2/5/18 8:11 PM:
--------------------------------------------------------
Hi [~Steven Rand], I made it work by using following resource requests:
{code}
ResourceRequest nodeRequest =
createResourceRequest(GB, node1.getHostName(), 1, 4, true);
ResourceRequest rackRequest =
createResourceRequest(GB, node1.getRackName(), 1, 4, true);
ResourceRequest anyRequest =
createResourceRequest(GB, ResourceRequest.ANY, 1, 4, true);
...
verifyPreemption(4, 4);
{code}
Does this sound good to you?
BTW, there is one extra empty line after method setAllAMContainersOnNode.
was (Author: yufeigu):
Hi [~Steven Rand], I made it work by using following resource requests:
{code}
ResourceRequest nodeRequest =
createResourceRequest(GB, node1.getHostName(), 1, 4, true);
ResourceRequest rackRequest =
createResourceRequest(GB, node1.getRackName(), 1, 4, true);
ResourceRequest anyRequest =
createResourceRequest(GB, ResourceRequest.ANY, 1, 4, true);
...
verifyPreemption(4, 4);
{code}
Does this sound good to you?
> avoid AM preemption caused by RRs for specific nodes or racks
> -------------------------------------------------------------
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 3.0.0
> Reporter: Steven Rand
> Assignee: Steven Rand
> Priority: Major
> Attachments: YARN-7655-001.patch, YARN-7655-002.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
