Hi Brock Thanks for a prompt and to-the-point response. It is working as you said.
Best, Vivek -- On Tue, Sep 20, 2011 at 6:25 PM, Brock Noland <[email protected]> wrote: > Hi, > > On Tue, Sep 13, 2011 at 12:27 PM, Vivek K <[email protected]> wrote: > > Hi all, > > > > I am trying to build a Hadoop/MR application in c++ using hadoop-pipes. I > > have been able to successfully work with my own mappers and reducers, but > > now I need to generate output (from reducer) in a format different from > the > > default TextOutputFormat. I have a few questions: > > > > (1) Similar to Hadoop streaming, is there an option to set OutputFormat > in > > HadoopPipes (in order to use say > org.apache.hadoop.io.SequenceFile.Writer) ? > > I am using Hadoop version 0.20.2. > > > > (2) For a simple test on how to use an in-built non-default writer, I > tried > > the following: > > > > hadoop pipes -D hadoop.pipes.java.recordreader=true -D > > hadoop.pipes.java.recordwriter=false -input input.seq -output output > > -inputformat org.apache.hadoop.mapred.SequenceFileInputFormat -writer > > org.apache.hadoop.io.SequenceFile.Writer -program my_test_program > > > -writer wants an outputformat: > > if (results.hasOption("writer")) { > setIsJavaRecordWriter(job, true); > job.setOutputFormat(getClass(results, "writer", job, > OutputFormat.class)); > > > > As such I think you want: > > -writer org.apache.hadoop.mapred.SequenceFileOutputFormat > > SequenceFile.Writer simply writes sequence files has nothing todo with > MapReduce. > > This is also wrong: > > hadoop.pipes.java.recordwriter=false > > Brock >
