[
https://issues.apache.org/jira/browse/HADOOP-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12637593#action_12637593
]
Zheng Shao commented on HADOOP-4360:
------------------------------------
The array of completion events at the JobTracker is correct - there is no
duplicate. I didn't try that -events but I have a web interface to view the
completed events on the job tracker.
The problem is only at the TaskTracker. However, different TaskTrackers have
different patterns of duplicates.
A lot of them (but not all) are 100 duplicates at a time (1,2,3...,100,
1,2,3,...100), which also signals the problem is at the communication protocol
(where we poll 100 events at a time).
> Reducers hang in SHUFFLING phase due to duplicate completed tasks in
> TaskTracker.FetchStatus.allMapEvents
> ---------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4360
> URL: https://issues.apache.org/jira/browse/HADOOP-4360
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2
> Reporter: Zheng Shao
>
> On our cluster we have seen JobTracker went to a weird state that a lot of
> TaskTrackers are getting duplicate entries in
> TaskTracker.FetchStatus.allMapEvents.
> Since TaskTracker fetches new completed map tasks using the size of the
> allMapEvents as starting index, this prohibits the tasktracker from getting
> all completed map tasks. And as a result, reducer just hangs in the shuffling
> status.
> The problem does not get fixed by killing and restarting TaskTracker, and
> when it happens a lot of TaskTrackers will show the same problem.
> It seems some problems happen to the communication between JobTracker and
> TaskTracker.
> An easy preventive fix will be to include the starting index of the list of
> completed map tasks from JobTracker to TaskTracker, so that TaskTracker can
> just throw away the data if the starting index does not match the current
> size of the array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
