[
https://issues.apache.org/jira/browse/SPARK-16944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411315#comment-15411315
]
Michael Gummelt commented on SPARK-16944:
-----------------------------------------
Yea, we typically call it "delay scheduling". It was first written about by
the Spark/Mesos researchers:
http://elmeleegy.com/khaled/papers/delay_scheduling.pdf
Spark already has `spark.locality.wait`, but that's how long the task scheduler
will wait until an executor will come up with the preferred locality. We need
a similar concept for waiting for offers to come in so we can place the
executor correctly in the first place.
> [MESOS] Improve data locality when launching new executors when dynamic
> allocation is enabled
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-16944
> URL: https://issues.apache.org/jira/browse/SPARK-16944
> Project: Spark
> Issue Type: New Feature
> Components: Mesos
> Affects Versions: 2.0.0
> Reporter: Sun Rui
>
> Currently Spark on Yarn supports better data locality by considering the
> preferred locations of the pending tasks when dynamic allocation is enabled.
> Refer to https://issues.apache.org/jira/browse/SPARK-4352. It would be better
> that Mesos can also support this feature.
> I guess that some logic existing in Yarn could be reused by Mesos.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
