[ http://issues.apache.org/jira/browse/HADOOP-76?page=comments#action_12441868 ] Owen O'Malley commented on HADOOP-76: -------------------------------------
I think it is better for now to just use JobConf.setSpeculativeExecution for both maps and reduces. machinesWhereFailed is a list of machines where the task has failed. To find where it is currently running, you need to use recentTasks. Currently, the recentTasks is a set of task ids that are running. You should probably make it a map from task id to task tracker id. Don't block speculative reduces based on other reduces running. That would make a perpetually busy cluster never run speculative reduces. *** Please create a library that lets you create files off to the side and when you call commit moves them into place. Speculative reduces need it, but that functionality is useful other places, such as side effect-based maps. The class should also have an abort method that cleans up. The defaults for the map speculative execution don't look too unreasonable, so just use them for now. > Implement speculative re-execution of reduces > --------------------------------------------- > > Key: HADOOP-76 > URL: http://issues.apache.org/jira/browse/HADOOP-76 > Project: Hadoop > Issue Type: Improvement > Components: mapred > Affects Versions: 0.1.0 > Reporter: Doug Cutting > Assigned To: Sanjay Dahiya > Priority: Minor > Attachments: spec_reducev.patch > > > As a first step, reduce task outputs should go to temporary files which are > renamed when the task completes. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
