[
https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14645333#comment-14645333
]
Saisai Shao commented on SPARK-4352:
------------------------------------
Hi [~tgraves], to be clear the original way of setting locality preferences
through SparkContext in Yarn mode is invalid long time ago, also this allowing
user to specify the locality preferences is not a intuitive and optimal way,
especially when dynamic allocation is introduced.
This proposal tries to calculate the optimal locality preferences of containers
through pending stages' inputs, this locality preferences is calculated in the
run-time, and will be changed through dynamic allocation. This works fine with
dynamic allocation compared to previous way.
For the previous way of allowing user to specify the locality preference, I'm
not sure do we need to support this again, if this function is still required I
think I could combine it to the new way to support it again.
> Incorporate locality preferences in dynamic allocation requests
> ---------------------------------------------------------------
>
> Key: SPARK-4352
> URL: https://issues.apache.org/jira/browse/SPARK-4352
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 1.2.0
> Reporter: Sandy Ryza
> Assignee: Saisai Shao
> Priority: Critical
> Fix For: 1.5.0
>
> Attachments: Supportpreferrednodelocationindynamicallocation.pdf
>
>
> Currently, achieving data locality in Spark is difficult unless an
> application takes resources on every node in the cluster.
> preferredNodeLocalityData provides a sort of hacky workaround that has been
> broken since 1.0.
> With dynamic executor allocation, Spark requests executors in response to
> demand from the application. When this occurs, it would be useful to look at
> the pending tasks and communicate their location preferences to the cluster
> resource manager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
