[ https://issues.apache.org/jira/browse/HADOOP-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633903#action_12633903 ]
Arun C Murthy commented on HADOOP-3136: --------------------------------------- Ok, with HADOOP-249 getting in it's imperative for us to consider jvm-reuse in the scheduling algorithms too - it really behooves us to think much harder about the _global_ scheduling problem and it probably requires a prototype or two to really get under the skin of the beast. I'll go ahead and open a new jira for that. Meanwhile, given the benefits of assigning multiple tasks, I'd propose we go ahead and do a quick-fix along the lines suggested by Runping/Eric. It clearly is a win over what we have currently. Thoughts? > Assign multiple tasks per TaskTracker heartbeat > ----------------------------------------------- > > Key: HADOOP-3136 > URL: https://issues.apache.org/jira/browse/HADOOP-3136 > Project: Hadoop Core > Issue Type: Improvement > Components: mapred > Reporter: Devaraj Das > Assignee: Arun C Murthy > Fix For: 0.20.0 > > Attachments: HADOOP-3136_0_20080805.patch, > HADOOP-3136_1_20080809.patch, HADOOP-3136_2_20080911.patch > > > In today's logic of finding a new task, we assign only one task per heartbeat. > We probably could give the tasktracker multiple tasks subject to the max > number of free slots it has - for maps we could assign it data local tasks. > We could probably run some logic to decide what to give it if we run out of > data local tasks (e.g., tasks from overloaded racks, tasks that have least > locality, etc.). In addition to maps, if it has reduce slots free, we could > give it reduce task(s) as well. Again for reduces we could probably run some > logic to give more tasks to nodes that are closer to nodes running most maps > (assuming data generated is proportional to the number of maps). For e.g., if > rack1 has 70% of the input splits, and we know that most maps are data/rack > local, we try to schedule ~70% of the reducers there. > Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.