Hi, I tried but it didnt work.
I don't understand why not it works, I only want that the first reducer write my object into DHFS and the second mapper reads this object from DHFS. I'm try to write object with SequenceFileOutFormat and I've have my own Writable, obviously my object implements Writable, but I continues doesn't work, and I also put job.setOutputFormatClass(SequenceFileOutputFormat.class) and SequenceFileOutputFormat.setOutputPath(conf, outputDir). However, I'm not using "setOutputCompression". Joan 2011/1/18 David Rosenstrauch <[email protected]> > I assumed you were already doing this but yes, Alain is correct, you need > to set the output format too. > > I initialize writing to sequence files like so: > > job.setOutputFormatClass(SequenceFileOutputFormat.class); > FileOutputFormat.setOutputName(job, dataSourceName); > FileOutputFormat.setOutputPath(job, hdfsJobOutputPath); > FileOutputFormat.setCompressOutput(job, true); > FileOutputFormat.setOutputCompressorClass(job, DefaultCodec.class); > SequenceFileOutputFormat.setOutputCompressionType(job, > SequenceFile.CompressionType.BLOCK); > > DR > > > > On 01/14/2011 01:27 PM, MONTMORY Alain wrote: > >> Hi, >> >> I think you have to put : >> job.setOutputFormatClass(SequenceFileOutputFormat.class); >> to make it works.. >> hopes this help >> >> Alain >> >> [@@THALES GROUP RESTRICTED@@] >> >> De : Joan [mailto:[email protected]] >> Envoyé : vendredi 14 janvier 2011 13:58 >> À : mapreduce-user >> Objet : how to write custom object using M/R >> >> Hi, >> >> I'm trying to write (K,V) where K is a Text object and V's CustomObject. >> But It doesn't run. >> >> I'm configuring output job like: SequenceFileInputFormat so I have job >> with: >> >> job.setMapOutputKeyClass(Text.class); >> job.setMapOutputValueClass(CustomObject.class); >> job.setOutputKeyClass(Text.class); >> job.setOutputValueClass(CustomObject.class); >> >> SequenceFileOutputFormat.setOutputPath(job, new Path("myPath"); >> >> And I obtain the next output (this is a file: part-r-00000): >> >> K CustomObject@2b237512 >> K CustomObject@24db06de >> ... >> >> When this job finished I run other job which input is >> SequenceFileInputFormat but It doesn't run: >> >> The configuration's second job is: >> >> job.setInputFormatClass(SequenceFileInputFormat.class); >> SequenceFileInputFormat.addInputPath(job, new Path("myPath")); >> >> But I get an error: >> >> java.io.IOException: hdfs://localhost:30000/user/hadoop/out/part-r-00000 >> not a SequenceFile >> at >> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1523) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1483) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1451) >> at >> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1432) >> at >> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:60) >> >> >> Can someone help me? Because I don't understand it. I don't know to save >> my object in first M/R and how to use it in second M/R >> >> Thanks >> >> Joan >> >> >> >> >
