[
https://issues.apache.org/jira/browse/SPARK-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14051575#comment-14051575
]
Mridul Muralidharan commented on SPARK-2277:
--------------------------------------------
I have not rechecked that the code, but the way it was originally written by me
was :
a) Task preference is decoupled from availability of the node.
For example, we need not have an executor on a host for which a block has host
preference (example dfs blocks on a shared cluster)
Also note that a block might have one or more preferred location.
b) We lookup the rack for the preferred location to get preferred rack.
As with (a), there need not be an executor on that rack. This is just the rack
preference.
c) At schedule time, for an executor, we lookup the host/rack of the executors
location - and decide appropriately based on that.
In this context, I think your requirement is already handled.
Even if we dont have any hosts alive on a rack, those tasks would still be
mentioned with rack local preference in task set manager.
When an executor comes in (existing or new), we check that executors rack with
task preference - and it would now be marked rack local.
> Make TaskScheduler track whether there's host on a rack
> -------------------------------------------------------
>
> Key: SPARK-2277
> URL: https://issues.apache.org/jira/browse/SPARK-2277
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Rui Li
>
> When TaskSetManager adds a pending task, it checks whether the tasks's
> preferred location is available. Regarding RACK_LOCAL task, we consider the
> preferred rack available if such a rack is defined for the preferred host.
> This is incorrect as there may be no alive hosts on that rack at all.
> Therefore, TaskScheduler should track the hosts on each rack, and provides an
> API for TaskSetManager to check if there's host alive on a specific rack.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
