[ http://issues.apache.org/jira/browse/HADOOP-362?page=comments#action_12422786 ] Owen O'Malley commented on HADOOP-362: --------------------------------------
There is a another aspect/cause of this that we just observed. What we saw was a large job with 5 maps that were not running (all maps were done) and yet all of the reduces were waiting for their output. Closer examination of the log showed that the maps had been logged as complete by the job tracker and were not being run anywhere. The TaskInProgress was showing 100% complete in the web ui. The Task details however was showing 50% and running. My best guess as to the failure scenario is: 1. task tracker sends progress of (50%, RUNNING) for map_123 to job tracker 2. task tracker gets time out on progress message 3. map_123 finishes 4. task tracker send progress of (100%, SUCCEED) for map_123 to job tracker 5. the two messages are taken from the rpc queue and given to separate handler threads 6. the SUCCEED message thread gets the JobTracker lock and updates the status of the Task and TaskInProgress. 7. the RUNNING message thread gets the lock and updates the status of the Task 8. reduces ask for the map output and nothing is available Therefore, we also need to make sure that Tasks are not allowed to move from SUCCEED or FAILED to RUNNING. That will solve _this_ problem. However, this represents a much deeper and pervasive problem that we will need to address that any two rpc calls from the _same_ thread can be executed in an arbitrary order. > tasks can get lost when reporting task completion to the JobTracker has an > error > -------------------------------------------------------------------------------- > > Key: HADOOP-362 > URL: http://issues.apache.org/jira/browse/HADOOP-362 > Project: Hadoop > Issue Type: Bug > Components: mapred > Reporter: Devaraj Das > Assigned To: Owen O'Malley > Attachments: lost-status-updates.patch > > > Basically, the JobTracker used to lose some updates about successful map > tasks and it would assume that the tasks are still running (the old progress > report is what it used to display in the web page). Now this would cause the > reduces to also wait for the map output and they would never receive the > output. This would cause the job to appear as if it was hung. > > The following piece of code sends the status of tasks to the JobTracker: > > synchronized (this) { > for (Iterator it = runningTasks.values().iterator(); > it.hasNext(); ) { > TaskInProgress tip = (TaskInProgress) it.next(); > TaskStatus status = tip.createStatus(); > taskReports.add(status); > if (status.getRunState() != TaskStatus.RUNNING) { > if (tip.getTask().isMapTask()) { > mapTotal--; > } else { > reduceTotal--; > } > it.remove(); > } > } > } > > // > // Xmit the heartbeat > // > > TaskTrackerStatus status = > new TaskTrackerStatus(taskTrackerName, localHostname, > httpPort, taskReports, > failures); > int resultCode = jobClient.emitHeartbeat(status, justStarted); > > > Notice that the completed TIPs are removed from runningTasks data structure. > Now, if the emitHeartBeat threw an exception (if it could not communicate > with the JobTracker till the IPC timeout expires) then this update is lost. > And the next time it sends the hearbeat this completed task's status is > missing and hence the JobTracker doesn't know about this completed task. So, > one solution to this is to remove the completed TIPs from runningTasks after > emitHeartbeat returns. Here is how the new code would look like: > > > synchronized (this) { > for (Iterator it = runningTasks.values().iterator(); > it.hasNext(); ) { > TaskInProgress tip = (TaskInProgress) it.next(); > TaskStatus status = tip.createStatus(); > taskReports.add(status); > } > } > > // > // Xmit the heartbeat > // > > TaskTrackerStatus status = > new TaskTrackerStatus(taskTrackerName, localHostname, > httpPort, taskReports, > failures); > int resultCode = jobClient.emitHeartbeat(status, justStarted); > synchronized (this) { > for (Iterator it = runningTasks.values().iterator(); > it.hasNext(); ) { > TaskInProgress tip = (TaskInProgress) it.next(); > if (tip.runstate != TaskStatus.RUNNING) { > if (tip.getTask().isMapTask()) { > mapTotal--; > } else { > reduceTotal--; > } > it.remove(); > } > } > } > -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
