[
https://issues.apache.org/jira/browse/YARN-7655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328166#comment-16328166
]
Yufei Gu commented on YARN-7655:
--------------------------------
Do several AMs end up running on a limited number of NM? If the answer is Yes,
what I am guessing is that Spark's dynamic allocation and {{assignmultiple}} of
FS work together to make that. With Spark's dynamic allocation, an app may only
have one AM container for a while before it requires tasks containers(That's
what I am guessing though, I'm not familiar with Spark). Several AMs may be
allocated into one or very few NMs while assignmultiple is on. This also aligns
with -1 maxAMShare in your setting.
If that so, YARN-1042(anti-affinity) would help, but it isn't done yet. The
MAPREDUCE-6871 provide a hacky way to do that as well. You can specify
nodes/racks for AMs, so you can spread out AMs. This feature is not only for
MR, Spark AM did the similar implementation.
> avoid AM preemption caused by RRs for specific nodes or racks
> -------------------------------------------------------------
>
> Key: YARN-7655
> URL: https://issues.apache.org/jira/browse/YARN-7655
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 3.0.0
> Reporter: Steven Rand
> Assignee: Steven Rand
> Priority: Major
> Attachments: YARN-7655-001.patch
>
>
> We frequently see AM preemptions when
> {{starvedApp.getStarvedResourceRequests()}} in
> {{FSPreemptionThread#identifyContainersToPreempt}} includes one or more RRs
> that request containers on a specific node. Since this causes us to only
> consider one node to preempt containers on, the really good work that was
> done in YARN-5830 doesn't save us from AM preemption. Even though there might
> be multiple nodes on which we could preempt enough non-AM containers to
> satisfy the app's starvation, we often wind up preempting one or more AM
> containers on the single node that we're considering.
> A proposed solution is that if we're going to preempt one or more AM
> containers for an RR that specifies a node or rack, then we should instead
> expand the search space to consider all nodes. That way we take advantage of
> YARN-5830, and only preempt AMs if there's no alternative. I've attached a
> patch with an initial implementation of this. We've been running it on a few
> clusters, and have seen AM preemptions drop from double-digit occurrences on
> many days to zero.
> Of course, the tradeoff is some loss of locality, since the starved app is
> less likely to be allocated resources at the most specific locality level
> that it asked for. My opinion is that this tradeoff is worth it, but
> interested to hear what others think as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
