[
http://issues.apache.org/jira/browse/HADOOP-76?page=comments#action_12445173 ]
Sanjay Dahiya commented on HADOOP-76:
-------------------------------------
Looking at FileSystem implementation here is a concern
FileSystem is an abstract class, extended by Local/Distributed FileSystem.
ideal way to have a PhasedFileSystem would be if FileSystem was an interface,
we could implement it and take a FileSystem in the constructor of
PhasedFIleSystem. some thing like -
PhasedFileSystem implements FileSystem{
public PhasedFileSystem(FileSysten fs);
// channel all methods to fs
// implement commit() / abort()
}
In this case we could use Phased functionality with both Local/Distributed
FileSystem, still maintaining interface compatibility. Doing this with base
abstract class will cause some dummy objects. as in next option
Next option is to have something like
PhasedFileSystem extends FileSystem{
private FileSystem fs ;
protected PhasedFileSystem(FileSysten fs) {
super(fs.getConf()); // not used, just dummy for base class creation
this.fs = fs ;
}
protected PhasedFileSystem(conf){ super(conf); throw NotSupportedException;
}
public static getNamed(...) {
return new PhasedFileSystem(FileSystem.getNamed..) ;
}
// All other methods channel calls to fs
commit() ;
abort();
}
Last option is to add extra methods to FileSystem itself, which will not be
used in most cases but will be available in derived FileSystems. This doesnt
sound too good.
Option 2 is something that will work well for us, even though its not a good
design.
Comments?
> Implement speculative re-execution of reduces
> ---------------------------------------------
>
> Key: HADOOP-76
> URL: http://issues.apache.org/jira/browse/HADOOP-76
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.1.0
> Reporter: Doug Cutting
> Assigned To: Sanjay Dahiya
> Priority: Minor
> Attachments: Hadoop-76.patch, spec_reducev.patch
>
>
> As a first step, reduce task outputs should go to temporary files which are
> renamed when the task completes.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
