[
https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094872#comment-14094872
]
Sandy Ryza commented on SPARK-2089:
-----------------------------------
My opinion is that we should have a narrower API for specifying the preferred
node data. Ultimately we just want to give Spark enough information to be able
to make location requests to cluster schedulers. If we want to do fancier
things with additional data that InputSplits provide, we can add utility
methods for converting that data to location preferences. The
YarnAllocationHandler shouldn't need this data to make requests at runtime.
Here's a proposal for a cleaner API here:
We add a Spark property spark.scheduler.taskLocationsPreferences. The value is
a set of "task locations preferences", i.e. a set of nodes and racks where we
know we will run a single task. The string format would look something like
"(node1|node2|rack1),(node4|node2|rack2|rack1)".
We add a an API taskLocationPreferences(inputFormatClass: Class[InputFormat],
path: Path, conf: Configuration): String
[~tgraves] [~mridula] [~pwendell] what do you think of this?
> With YARN, preferredNodeLocalityData isn't honored
> ---------------------------------------------------
>
> Key: SPARK-2089
> URL: https://issues.apache.org/jira/browse/SPARK-2089
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.0.0
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
> Priority: Critical
>
> When running in YARN cluster mode, apps can pass preferred locality data when
> constructing a Spark context that will dictate where to request executor
> containers.
> This is currently broken because of a race condition. The Spark-YARN code
> runs the user class and waits for it to start up a SparkContext. During its
> initialization, the SparkContext will create a YarnClusterScheduler, which
> notifies a monitor in the Spark-YARN code that . The Spark-Yarn code then
> immediately fetches the preferredNodeLocationData from the SparkContext and
> uses it to start requesting containers.
> But in the SparkContext constructor that takes the preferredNodeLocationData,
> setting preferredNodeLocationData comes after the rest of the initialization,
> so, if the Spark-YARN code comes around quickly enough after being notified,
> the data that's fetched is the empty unset version. The occurred during all
> of my runs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
