[jira] [Created] (SPARK-1524) TaskSetManager'd better not schedule tasks which has no preferred executorId using PROCESS_LOCAL in the first search process

[YanTang Zhai (JIRA)]

YanTang Zhai created SPARK-1524:
-----------------------------------

             Summary: TaskSetManager'd better not schedule tasks which has no 
preferred executorId using PROCESS_LOCAL in the first search process
                 Key: SPARK-1524
                 URL: https://issues.apache.org/jira/browse/SPARK-1524
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
            Reporter: YanTang Zhai
            Priority: Minor


ShuffleMapTask is constructed with TaskLocation which has only host not (host, 
executorID) pair in DAGScheduler.
When TaskSetManager schedules ShuffleMapTask which has no preferred executorId 
using specific execId host and PROCESS_LOCAL locality level, no tasks match the 
given locality constraint in the first search process.
We also find that the host used by Scheduler is hostname while the host used by 
TaskLocation is IP in our cluster. The tow hosts do not match, that makes 
pendingTasksForHost HashMap empty and the finding task process against our 
expectation.



--
This message was sent by Atlassian JIRA
(v6.2#6252)