[
https://issues.apache.org/jira/browse/SPARK-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972864#comment-13972864
]
Mridul Muralidharan commented on SPARK-1524:
--------------------------------------------
The expectation is to fallback to a previous schedule type in case the higher
level is not valid : though this is tricky in general case.
Will need to take a look at it - though given that I am tied up with other
things, if someone else wants to take a crack, please feel free to do so !
Btw, use of IP's and multiple hostnames for a host is not supported in spark -
so that is something that will need to be resolved at the deployment end.
> TaskSetManager'd better not schedule tasks which has no preferred executorId
> using PROCESS_LOCAL in the first search process
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-1524
> URL: https://issues.apache.org/jira/browse/SPARK-1524
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: YanTang Zhai
> Priority: Minor
>
> ShuffleMapTask is constructed with TaskLocation which has only host not
> (host, executorID) pair in DAGScheduler.
> When TaskSetManager schedules ShuffleMapTask which has no preferred
> executorId using specific execId host and PROCESS_LOCAL locality level, no
> tasks match the given locality constraint in the first search process.
> We also find that the host used by Scheduler is hostname while the host used
> by TaskLocation is IP in our cluster. The tow hosts do not match, that makes
> pendingTasksForHost HashMap empty and the finding task process against our
> expectation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
