[
https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandy Ryza updated SPARK-4352:
------------------------------
Description:
Currently, achieving data locality in Spark is difficult unless an application
takes resources on every node in the cluster. preferredNodeLocalityData
provides a sort of hacky workaround that has been broken since 1.0.
With dynamic executor allocation, Spark requests executors in response to
demand from the application. When this occurs, it would be useful to look at
the pending tasks and communicate their location preferences to the cluster
resource manager.
was:
Currently, achieving data locality in Spark is difficult u
preferredNodeLocalityData provides a sort of hacky workaround that has been
broken since 1.0.
> Incorporate locality preferences in dynamic allocation requests
> ---------------------------------------------------------------
>
> Key: SPARK-4352
> URL: https://issues.apache.org/jira/browse/SPARK-4352
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 1.2.0
> Reporter: Sandy Ryza
>
> Currently, achieving data locality in Spark is difficult unless an
> application takes resources on every node in the cluster.
> preferredNodeLocalityData provides a sort of hacky workaround that has been
> broken since 1.0.
> With dynamic executor allocation, Spark requests executors in response to
> demand from the application. When this occurs, it would be useful to look at
> the pending tasks and communicate their location preferences to the cluster
> resource manager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
