On Feb 14, 2008, at 2:09 PM, Jason Venner wrote:
We write a separate file in many our our mappers and or reducers.
We are somewhat concerned about speculative execution and what
happens to the output files of killed jobs, but it seems to work fine.
We build the output files by passing in a *tag* which is used as
the last part of the file name
Path destinationFile = new Path( conf.get( "mapred.output.dir" ),
tag );
The _blessed_ way to create tasks' side-effect files:
http://hadoop.apache.org/core/docs/r0.16.0/mapred_tutorial.html#Task
+Side-Effect+Files
Arun
Lukas Vlcek wrote:
Hi,
Is it possible to have Reducer output the data into two different
formats at
the same time?
For example one output in SequenceFileOutputFormat for further
processing by
consequential M/R job and second output in TextOutputFormat for
later human
review. Isn't this use case disqualified by the fact that there is
no way
how to set two output paths in JobConf?
I think it should be possible to create output file in
TextOutputFormat from
SequenceFileOutputFormat input file but this requires running of
extra M/R
job.
Regards,
Lukas
