[ 
http://issues.apache.org/jira/browse/HADOOP-76?page=comments#action_12445048 ] 
            
Owen O'Malley commented on HADOOP-76:
-------------------------------------

The PhasedRecordWriter won't handle all of the cases, because RecordWriters can 
write multiple files. Furthermore, they are user code and it would be better to 
minimize required changes to them.

A better approach would be to have a PhasedFileSystem that takes a base 
FileSystem and use that to commit/abort the changes. Then the framework could 
pass the PhasedFileSystem to the createRecordReader call and it would catch all 
of the files that the RecordWriter created. When the PhasedFileSystem gets a 
create call, it creates it in the base FileSystem with a mutated name. When the 
changes are commited, the files are all renamed. If the changes are aborted, 
the mutated filenames are deleted.



> Implement speculative re-execution of reduces
> ---------------------------------------------
>
>                 Key: HADOOP-76
>                 URL: http://issues.apache.org/jira/browse/HADOOP-76
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.1.0
>            Reporter: Doug Cutting
>         Assigned To: Sanjay Dahiya
>            Priority: Minor
>         Attachments: Hadoop-76.patch, spec_reducev.patch
>
>
> As a first step, reduce task outputs should go to temporary files which are 
> renamed when the task completes.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: 
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Reply via email to