[
https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569623#comment-14569623
]
Sandy Ryza commented on SPARK-4352:
-----------------------------------
In the case where the task number <= executor number * cores, I think my
earlier argument still stands. Any executor requests beyond the ones needed to
satisfy our preferences should be submitted with locality preferences. This
means we will be less likely to bunch up requests on particular nodes where
executors are not needed. Consider the extreme case where we want to request
100 executors but only have a single task with locality preferences, for data
on 3 nodes. Going purely by the ratio approach, we would end up requesting all
100 executors on those three nodes.
For the other cases, your approach makes sense to me.
> Incorporate locality preferences in dynamic allocation requests
> ---------------------------------------------------------------
>
> Key: SPARK-4352
> URL: https://issues.apache.org/jira/browse/SPARK-4352
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 1.2.0
> Reporter: Sandy Ryza
> Assignee: Saisai Shao
> Priority: Critical
> Attachments: Supportpreferrednodelocationindynamicallocation.pdf
>
>
> Currently, achieving data locality in Spark is difficult unless an
> application takes resources on every node in the cluster.
> preferredNodeLocalityData provides a sort of hacky workaround that has been
> broken since 1.0.
> With dynamic executor allocation, Spark requests executors in response to
> demand from the application. When this occurs, it would be useful to look at
> the pending tasks and communicate their location preferences to the cluster
> resource manager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
