dynamic heartbeat interval for the locality-aware task scheduling
-----------------------------------------------------------------
Key: HADOOP-3674
URL: https://issues.apache.org/jira/browse/HADOOP-3674
Project: Hadoop Core
Issue Type: Wish
Components: mapred
Reporter: Leitao Guo
Priority: Minor
In current hadoop release (0.17.0), there is no special scheduling policy for
those tasktrackers who have no data for some jobs. So, there would be
inefficient in some senarios. For example, tasktracker A has the data for a
job, but tasktracker B, which has no data for this job, sends the heartbeat
message to the jobtracker for a new task before tasktrack A. The task may be
scheduled to B instead of A. While Jobtracker has to find a new task for
tasktracker A when A ask for a new task.
In this situation, if jobtracker has some reservation policy, such as reserve
the task for tasktracker A and let B ask for new task in the next heartbeat
message, that would be more efficient. Because before tasktracker B asking for
new task the second time, tasktracker A has applied for a new task and
jobtracker has scheduled the task to A.
Here is a rough idea to deal with the senario above:
(1) Jobtracker receives the heartbeat message sent by tasktracker B, which has
no data for any job.
(2) Jobtracker send response message to tasktracker B with a new heartbeat
message interval, but does not schedule new task to B. The new heartbeat
interval should be shorter the current heartbeat interval, for example,
current_heartbeat_interval/2.
(3) Tasktracker B receive the response from jobtracker, and sends another
heartbeat message for a new task after a period of current_heartbeat_interval/2
.
(4) Jobtracker then find a new task for tasktracker B.
This is just an primary idea for the improvement of the locality-aware
scheduling. Any comments are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
