[ https://issues.apache.org/jira/browse/SPARK-2277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14050886#comment-14050886 ]
Mridul Muralidharan commented on SPARK-2277: -------------------------------------------- I am not sure I follow this requirement. For preferred locations, we populate their corresponding racks (if available) as preferred rack. For available executors hosts, we lookup the rack they belong to - and then see if that rack is preferred or not. This, ofcourse, assumes a host is only on a single rack. What exactly is the behavior you are expecting from scheduler ? > Make TaskScheduler track whether there's host on a rack > ------------------------------------------------------- > > Key: SPARK-2277 > URL: https://issues.apache.org/jira/browse/SPARK-2277 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.0.0 > Reporter: Rui Li > > When TaskSetManager adds a pending task, it checks whether the tasks's > preferred location is available. Regarding RACK_LOCAL task, we consider the > preferred rack available if such a rack is defined for the preferred host. > This is incorrect as there may be no alive hosts on that rack at all. > Therefore, TaskScheduler should track the hosts on each rack, and provides an > API for TaskSetManager to check if there's host alive on a specific rack. -- This message was sent by Atlassian JIRA (v6.2#6252)