We write a separate file in many our our mappers and or reducers. We are
somewhat concerned about speculative execution and what happens to the
output files of killed jobs, but it seems to work fine.
We build the output files by passing in a *tag* which is used as the
last part of the file name
Path destinationFile = new Path( conf.get( "mapred.output.dir" ), tag );
Lukas Vlcek wrote:
Hi,
Is it possible to have Reducer output the data into two different formats at
the same time?
For example one output in SequenceFileOutputFormat for further processing by
consequential M/R job and second output in TextOutputFormat for later human
review. Isn't this use case disqualified by the fact that there is no way
how to set two output paths in JobConf?
I think it should be possible to create output file in TextOutputFormat from
SequenceFileOutputFormat input file but this requires running of extra M/R
job.
Regards,
Lukas
