[
http://issues.apache.org/jira/browse/HADOOP-639?page=comments#action_12451982 ]
Arun C Murthy commented on HADOOP-639:
--------------------------------------
Ok, while we continue to track the *metrics* part of TaskTrackerStatus via
HADOOP-657 I propose we move forward on the 'lost messages' part over here...
Here are some thoughts (with due credits to Owen):
Define new classes:
TaskTrackerAction implements Writable {
byte actionId = 0;
// ...
}
KillJobAction extends TaskTrackerAction {
byte actionId = 1;
// ...
}
KillTaskAction extends TaskTrackerAction {
byte actionId = 2;
// ...
}
StartTaskAction extends TaskTrackerAction {
byte actionId = 3;
// ...
}
The distinction between the KillTaskAction & KillJobAction is done to fix
HADOOP-737 ...
Another class:
class JTResponse {
long seqNo; // explained below
List<TaskTrackerAction> actions;
}
The new api replacing
int emitHeartbeat(TaskTrackerStatus status, boolean initialContact) throws
IOException;
Task pollForNewTask(String trackerName) throws IOException;
String[] pollForTaskWithClosedJob(String trackerName) throws IOException;
is:
* JTResponse updateStatus(TaskTrackerStatus status, long ackNo) throws
IOException; *
Details about the seqNo/ackNo:
------------------------------------
The idea is that there is a feedback (seq/ack) mechanism between the JT & TT
which works as follows...
TT starts off by sending an ack of '-1' (indicates initial contact, replaces
the existing 'initialContact' boolean); and at every step the JT increments the
ack and sends a new JTResponse object with the incremented ack as the 'seqNo'
and 'actions'. The JT also stores the last seq and the JTResponse object sent
to each of the task-trackers. OTOH the TT also stores the last 'seq' which it
recieved from the JT, which is what it sends out in the subsequent heartbeat as
'ack'.
How does this help? If a TT misses the heartbeat response from the JT, it sends
a stale which 'ack' disagrees with the newer 'seq' on the JT, this prompts the
JT to resend the 'saved' JTResponse object back to the TT... thus solving the
'lost messages' issue. If JT never hears back from a TT for a long time the
existing ExpireTrackers.run removes the TT from its queue and also discards the
saved JTResponse object for that TT.
-*-*-
Thoughts?
> task cleanup messages can get lost, causing task trackers to keep tasks
> forever
> -------------------------------------------------------------------------------
>
> Key: HADOOP-639
> URL: http://issues.apache.org/jira/browse/HADOOP-639
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.7.2
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.9.0
>
>
> If the pollForTaskWithClosedJob call from a job tracker to a task tracker
> times out when a job completes, the tasks are never cleaned up. This can
> cause the mini m/r cluster to hang on shutdown, but also is a resource leak.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
