Just a question:

did we consider "speculative execution of reduces with sub-splitting"?
That is, for each of the last / slow reduce tasks run several task subslitting the input of the original task? The implementation would be probably more complicated, but the benefits may be huge -- currently, it is not uncommon to see the single last reduce task running for hours (if not for days).

-- ab

On Oct 19, 2006, at 10:43 AM, Sanjay Dahiya (JIRA) wrote:

     [ http://issues.apache.org/jira/browse/HADOOP-76?page=all ]

Sanjay Dahiya updated HADOOP-76:
--------------------------------

    Attachment: Hadoop-76.patch

This patch is up for review.

Here is the list of changes included in this patch -

Replaced recentTasks to a Map, added a new method in TaskInProgress hasRanOnMachine, which looks at this Map and hasFailedOnMachines(). This is used to avoid scheduling multiple reduce instances of same task on the same node.

Added a PhasedRecordWriter, which takes a RecordWriter, tempName, finalName. Another option was to create a PhasedOutputFormat, this seems more natural as it works with any existing OutputFormat and RecordWriter. Records are written to tempName and when commit is called they are moved to finalName.

ReduceTask.run() - if speculative execution is enabled then reduce output is written to a temp location using PhasedRecordWriter. After task finishes the output is written to a final location. If some other speculative instance finishes first then TaskInProgress.shouldCloseForClosedJob() returns true for the taskId. On TaskTracker the task is killed by Process.destroy() so cleanup code is in TaskTracker instead of Task. The cleanup of Maps happen in Conf, which is probably misplaced. We could refactor this part for both Map and Reduce and move cleanup code to some utility classes which, given a Map and Reduce task track the files generated and cleanup if needed.

Added an extra attribute in TaskInProgress - runningSpeculative, to avoid running more than one speculative instances of ReduceTask. Too many Reduce instances for same task could increase load on Map machines, this needs discussion. I can revert this change back to allow some other number of instances of Reduces (MAX_TASK_FAILURES?).

comments

Implement speculative re-execution of reduces
---------------------------------------------

                Key: HADOOP-76
                URL: http://issues.apache.org/jira/browse/HADOOP-76
            Project: Hadoop
         Issue Type: Improvement
         Components: mapred
   Affects Versions: 0.1.0
           Reporter: Doug Cutting
        Assigned To: Sanjay Dahiya
           Priority: Minor
        Attachments: Hadoop-76.patch, spec_reducev.patch


As a first step, reduce task outputs should go to temporary files which are renamed when the task completes.

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply via email to