[
https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568410#comment-14568410
]
Saisai Shao commented on SPARK-4352:
------------------------------------
Hi [~sandyr], I start to know your algorithm, your algorithm is trying to
fulfill the smallest task number request at first, say in your example about 7
executors, firstly you assigned 5 on <a, b, c, d> to fulfill request of node d,
and left 2 for <a, b, c> because the request number is not enough. So what is
your purpose of fulfill the smallest request at first? Since node d only
requires 1/9 requests, why not satisfy a, b firstly, obviously they will run
more tasks and need more locality.
Also in several cases, the task number is larger than the total cores
available, say like 1000 tasks, but only have 15 executor requests, for your
algorithm, without ratio consideration, how to handle this?
> Incorporate locality preferences in dynamic allocation requests
> ---------------------------------------------------------------
>
> Key: SPARK-4352
> URL: https://issues.apache.org/jira/browse/SPARK-4352
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 1.2.0
> Reporter: Sandy Ryza
> Assignee: Saisai Shao
> Priority: Critical
> Attachments: Supportpreferrednodelocationindynamicallocation.pdf
>
>
> Currently, achieving data locality in Spark is difficult unless an
> application takes resources on every node in the cluster.
> preferredNodeLocalityData provides a sort of hacky workaround that has been
> broken since 1.0.
> With dynamic executor allocation, Spark requests executors in response to
> demand from the application. When this occurs, it would be useful to look at
> the pending tasks and communicate their location preferences to the cluster
> resource manager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
