[
https://issues.apache.org/jira/browse/HADOOP-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631671#action_12631671
]
Joydeep Sen Sarma commented on HADOOP-3136:
-------------------------------------------
+10 on #2
it's not surprising at all that the initial proposal in this jira makes
locality worse. one of our concerns with scheduling right now is the awful
locality for small jobs that results from greedy allocation. to that extent -
would really welcome any attempt at global scheduling.
the runnable queue per TT sounds very much like
https://issues.apache.org/jira/browse/HADOOP-3136?focusedCommentId=12618922#action_12618922
the queues should also allow JT to make scheduling decisions for a bunch of
jobs at one time - instead of for the job as soon as it arrives. (this can be
based on how busy the cluster is - with a idle cluster scheduling decisions are
better made immediately - with a busy cluster - one can club things together
and then decide since the nodes anyway have work to do).
this is also an opportunity to attack JT scalability. why keep the queues at
the JT only? send them to the TT as well. That way - we don't have to send a
heartbeat each time a slot frees up - those can also be batched up by the TT
without incurring any idle time (again because of the runnable queues) and
should help JT bottlenecks.
there are a lot of nuts that can be cracked at once here - so might be worth
more thought.
> Assign multiple tasks per TaskTracker heartbeat
> -----------------------------------------------
>
> Key: HADOOP-3136
> URL: https://issues.apache.org/jira/browse/HADOOP-3136
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Arun C Murthy
> Fix For: 0.19.0
>
> Attachments: HADOOP-3136_0_20080805.patch,
> HADOOP-3136_1_20080809.patch, HADOOP-3136_2_20080911.patch
>
>
> In today's logic of finding a new task, we assign only one task per heartbeat.
> We probably could give the tasktracker multiple tasks subject to the max
> number of free slots it has - for maps we could assign it data local tasks.
> We could probably run some logic to decide what to give it if we run out of
> data local tasks (e.g., tasks from overloaded racks, tasks that have least
> locality, etc.). In addition to maps, if it has reduce slots free, we could
> give it reduce task(s) as well. Again for reduces we could probably run some
> logic to give more tasks to nodes that are closer to nodes running most maps
> (assuming data generated is proportional to the number of maps). For e.g., if
> rack1 has 70% of the input splits, and we know that most maps are data/rack
> local, we try to schedule ~70% of the reducers there.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
