[
https://issues.apache.org/jira/browse/MAPREDUCE-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783564#action_12783564
]
ZhuGuanyin commented on MAPREDUCE-1247:
---------------------------------------
We print the java jstack when it became fake lost tasktracker on hadoop version
0.19, and found:
7 times the heartbeat thread waiting the TaskTracker lock ( 5 times because of
taskCleanup thread hold for a long time, 2 times because of reduce sub jvm call
TaskTracker.getMapCompletionEvents())
4 times the heartbeat thread waiting for the TaskTracker.TaskInProgress lock (
3 times because of taskCleanup thread hold for a long time, 1 time because of
TaskLauncher hold for a long time)
2 times the heartbeat thread waiting for the AllocatorPerContext lock
The heartbeat thread should only answer for the live or death status of
tasktracker, but current implentition it has too many others things to do, we
should let the heartbeat thread only do what it has to do.
> Send out-of-band heartbeat to avoid fake lost tasktracker
> ---------------------------------------------------------
>
> Key: MAPREDUCE-1247
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1247
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: ZhuGuanyin
>
> Currently the TaskTracker report task status to jobtracker through heartbeat,
> sometimes if the tasktracker lock the tasktracker to do some cleanup job,
> like remove task temp data on disk, the heartbeat thread would hang for a
> long time while waiting for the lock, so the jobtracker just thought it had
> lost and would reschedule all its finished maps or un finished reduce on
> other tasktrackers, we call it "fake lost tasktracker", some times it doesn't
> acceptable especially when we run some large jobs. So We introduce a
> out-of-band heartbeat mechanism to send an out-of-band heartbeat in that case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
