[
https://issues.apache.org/jira/browse/SPARK-19373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889719#comment-15889719
]
Stavros Kontopoulos commented on SPARK-19373:
---------------------------------------------
[~mgummelt] +1 for task locality + dynamic allocation. That would mean also
decline offers if locality is not satisfied, until you get the appropriate
nodes? In other words, it means trying to optimize locality, while getting
random offers I guess...
> Mesos implementation of spark.scheduler.minRegisteredResourcesRatio looks at
> acquired cores rather than registerd cores
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19373
> URL: https://issues.apache.org/jira/browse/SPARK-19373
> Project: Spark
> Issue Type: Bug
> Components: Mesos
> Affects Versions: 1.6.3, 2.0.2, 2.1.0
> Reporter: Michael Gummelt
> Assignee: Michael Gummelt
> Fix For: 2.2.0
>
>
> We're currently using `totalCoresAcquired` to account for registered
> resources, which is incorrect. That variable measures the number of cores
> the scheduler has accepted. We should be using `totalCoreCount` like the
> other schedulers do.
> Fixing this is important for locality, since users often want to wait for all
> executors to come up before scheduling tasks to ensure they get a node-local
> placement.
> original PR to add support: https://github.com/apache/spark/pull/8672/files
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
