[
https://issues.apache.org/jira/browse/HADOOP-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631666#action_12631666
]
Arun C Murthy commented on HADOOP-3136:
---------------------------------------
Matei, I agree that assigning atleast one map and one reduce will help - at
least in the single reducer case. A good short-term goal.
bq. Beyond that, for maps, maybe it would also help to limit the number you
launch per heartbeat (say to 2)
This does have implications for the removal of the 'crazy heartbeats' (i.e.
currently the TaskTrackers send a heartbeat out as soon as _any_ task
completes). Maybe we should send a heartbeat immediately when a TaskTracker
notices that the _last running map_ completed?
bq. [...] or to ask the job for tasks until it stops giving you local tasks
We do need to be slightly careful here: if we do agree that 'global scheduling'
is the right approach then I'd rather not introduce short-term hacks to pull
tricks like those. I'm hoping to get everyone to think about a long-term
solution and keep that in mind while we target short-term fixes, does that make
sense?
> Assign multiple tasks per TaskTracker heartbeat
> -----------------------------------------------
>
> Key: HADOOP-3136
> URL: https://issues.apache.org/jira/browse/HADOOP-3136
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Arun C Murthy
> Fix For: 0.19.0
>
> Attachments: HADOOP-3136_0_20080805.patch,
> HADOOP-3136_1_20080809.patch, HADOOP-3136_2_20080911.patch
>
>
> In today's logic of finding a new task, we assign only one task per heartbeat.
> We probably could give the tasktracker multiple tasks subject to the max
> number of free slots it has - for maps we could assign it data local tasks.
> We could probably run some logic to decide what to give it if we run out of
> data local tasks (e.g., tasks from overloaded racks, tasks that have least
> locality, etc.). In addition to maps, if it has reduce slots free, we could
> give it reduce task(s) as well. Again for reduces we could probably run some
> logic to give more tasks to nodes that are closer to nodes running most maps
> (assuming data generated is proportional to the number of maps). For e.g., if
> rack1 has 70% of the input splits, and we know that most maps are data/rack
> local, we try to schedule ~70% of the reducers there.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
