[
https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576721#comment-14576721
]
Saisai Shao commented on SPARK-4352:
------------------------------------
Hi [~sandyr], I re-implement the allocation strategy according to the
discussion, now the new strategy will consider:
1. allocating new containers with with locality distribution ratio for locality
required tasks, and for locality-free tasks let Yarn to decide the location as
previous code.
2. Combine the existing executor/host distribution with new required containers
to better suit for dynamic allocation.
3. If dynamic allocation is not enabled, or preferred locality is empty, the
code logic is the same as previous code.
4. locality required executors will have the high priority to be allocated with
new container if current containers are not enough to match all the locality
required tasks.
Please help to review, any comment is greatly appreciated.
> Incorporate locality preferences in dynamic allocation requests
> ---------------------------------------------------------------
>
> Key: SPARK-4352
> URL: https://issues.apache.org/jira/browse/SPARK-4352
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 1.2.0
> Reporter: Sandy Ryza
> Assignee: Saisai Shao
> Priority: Critical
> Attachments: Supportpreferrednodelocationindynamicallocation.pdf
>
>
> Currently, achieving data locality in Spark is difficult unless an
> application takes resources on every node in the cluster.
> preferredNodeLocalityData provides a sort of hacky workaround that has been
> broken since 1.0.
> With dynamic executor allocation, Spark requests executors in response to
> demand from the application. When this occurs, it would be useful to look at
> the pending tasks and communicate their location preferences to the cluster
> resource manager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
