[
https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14980278#comment-14980278
]
Steve Loughran commented on SPARK-2089:
---------------------------------------
Bear in mind that if you start specifying locations, it takes longer for
placement requests to be granted. With the normal Yarn schedulers that's only
10+ seconds before it downgrades (if relax-placement==true), but it can impact
short-lived work.
For long-lived apps, placement policy is often driven by people wanting
anti-affinity in container placements, so that you get independent failure
patterns. YARN doesn't support this (and doesn't have it on its roadmap), so
people get to do it themselves. For the curious, SLIDER-82 is my homework there.
There's also "where I was last time", with our without fallback to
rest-of-cluster. Here the rapid fallback of YARN pushes long-lived services to
implement their own escalation/relaxation logic if they want to give YARN more
time to assign it to an explicit node.
Trying to do clever/complex placement is [a hard
problem|http://slider.apache.org/design/rolehistory.html]. I'd generally
advocating avoiding it. Having a list of nodes to request, and perhaps a list
of nodes to avoid, would be simpler. Maybe also add some option of placement
policy, with two: "none", "configured" for now, leaving room for future options
"anti-affinity", "historical", ... if someone ever wanted this stuff.
Oh, and have I mentioned testing all of this...
> With YARN, preferredNodeLocalityData isn't honored
> ---------------------------------------------------
>
> Key: SPARK-2089
> URL: https://issues.apache.org/jira/browse/SPARK-2089
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.0.0
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
> Priority: Critical
>
> When running in YARN cluster mode, apps can pass preferred locality data when
> constructing a Spark context that will dictate where to request executor
> containers.
> This is currently broken because of a race condition. The Spark-YARN code
> runs the user class and waits for it to start up a SparkContext. During its
> initialization, the SparkContext will create a YarnClusterScheduler, which
> notifies a monitor in the Spark-YARN code that . The Spark-Yarn code then
> immediately fetches the preferredNodeLocationData from the SparkContext and
> uses it to start requesting containers.
> But in the SparkContext constructor that takes the preferredNodeLocationData,
> setting preferredNodeLocationData comes after the rest of the initialization,
> so, if the Spark-YARN code comes around quickly enough after being notified,
> the data that's fetched is the empty unset version. The occurred during all
> of my runs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
