[
https://issues.apache.org/jira/browse/MAPREDUCE-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783574#action_12783574
]
ZhuGuanyin commented on MAPREDUCE-1247:
---------------------------------------
The taskCleanup thread lock the TaskTracker when it call
MapOutputFile.removeAll() through TaskTracker.purgeTask() to cleanup a task or
TaskTracker.purgeJob() to cleanup a job, if the midoutput file larger than
50GB, and there some other io operations on this disk, it would hold the
tasktracker lock for a long time enough to let the jobtracker treat this
tasktracker as dead.
I think the current heartbeat thread has to handle too many things which
doesn't its duty. the deadlock in tasktracker currently may still happen and
may not be found in current implentition. And I don't think it is the
hearbeat's duty to found the deadlock in tasktracker.
> Send out-of-band heartbeat to avoid fake lost tasktracker
> ---------------------------------------------------------
>
> Key: MAPREDUCE-1247
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1247
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: ZhuGuanyin
>
> Currently the TaskTracker report task status to jobtracker through heartbeat,
> sometimes if the tasktracker lock the tasktracker to do some cleanup job,
> like remove task temp data on disk, the heartbeat thread would hang for a
> long time while waiting for the lock, so the jobtracker just thought it had
> lost and would reschedule all its finished maps or un finished reduce on
> other tasktrackers, we call it "fake lost tasktracker", some times it doesn't
> acceptable especially when we run some large jobs. So We introduce a
> out-of-band heartbeat mechanism to send an out-of-band heartbeat in that case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
