[
https://issues.apache.org/jira/browse/HADOOP-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12567548#action_12567548
]
Amar Kamat commented on HADOOP-2014:
------------------------------------
bq. Uh, both are very expensive to do on every heartbeat (i.e. the inner loop)
isn't it?!
I should have mentioned it earlier. I was trying to link HADOOP-2119 and
HADOOP-2014 since HADOOP-2119 deals with {{JobInProgress.findNewTask()}}. The
algorithm mentioned above is for {{JobInProgress.findNewTask()}}. I got Owen's
point thats why I mentioned
{quote}
(and also some load/rack/io/map-slots considerations)
{quote}
Just wanted to make sure that we also do the same for _speculation_. From what
I followed, the discussion is mainly for {{Runnable && ~Running}}.
> Job Tracker should prefer input-splits from overloaded racks
> ------------------------------------------------------------
>
> Key: HADOOP-2014
> URL: https://issues.apache.org/jira/browse/HADOOP-2014
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Devaraj Das
>
> Currently, when the Job Tracker assigns a mapper task to a task tracker and
> there is no local split to the task tracker, the
> job tracker will find the first runable task in the mast task list and
> assign the task to the task tracker.
> The split for the task is not local to the task tracker, of course. However,
> the split may be local to other task trackers.
> Assigning the that task, to that task tracker may decrease the potential
> number of mapper attempts with data locality.
> The desired behavior in this situation is to choose a task whose split is not
> local to any task tracker.
> Resort to the current behavior only if no such task is found.
> In general, it will be useful to know the number of task trackers to which
> each split is local.
> To assign a task to a task tracker, the job tracker should first try to pick
> a task that is local to the task tracker and that has minimal number of task
> trackers to which it is local. If no task is local to the task tracker, the
> job tracker should try to pick a task that has minimal number of task
> trackers to which it is local.
> It is worthwhile to instrument the job tracker code to report the number of
> splits that are local to some task trackers.
> That should be the maximum number of tasks with data locality. By comparing
> that number with the the actual number of
> data local mappers launched, we can know the effectiveness of the job tracker
> scheduling.
> When we introduce rack locality, we should apply the same principle.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
