Good evening. Its monsoon madness here. The whole place is flooded. I hate this place.
I had posed this on IRC #hadoop before I went to sleep. I got this reply <riz0d> But when i try and increase it to multiple reducers ... I get multiple files part-XXXX in the output folder <riz0d> What is the best pattern of reading them 1. Loop through the directory and read one by one <riz0d> or Run another Map Reduce to do my manipulations on the data <riz0d> or 3. Use some reading method in hadoop(i am new here) to read the entire folder outputs <riz0d> The parts are saved in SequenceFileModel format <Toad> if you're running another mapreduce to build your model, you can point it at the directory and it will usually read all the part-* fine <Toad> if not, might as well loop through and do them one by one... though I think SequenceFileInputFormat can just take a glob path <Toad> like /foo/bar/part-* I tried passing the part-* into the SequenceFile.Reader. But it threw an exception. After searching for glob files i came across this FileStatus<http://hadoop.apache.org/core/docs/r0.16.4/api/org/apache/hadoop/fs/FileStatus.html> [] *globStatus<http://hadoop.apache.org/core/docs/r0.16.4/api/org/apache/hadoop/fs/FileSystem.html#globStatus%28org.apache.hadoop.fs.Path%29> *(Path<http://hadoop.apache.org/core/docs/r0.16.4/api/org/apache/hadoop/fs/Path.html> pathPattern) http://hadoop.apache.org/core/docs/r0.16.4/api/org/apache/hadoop/fs/FileSystem.html#globStatus(org.apache.hadoop.fs.Path) FileStatus.Path gives you the object. Pattern can be part-* Let me test it out. I will see what comes out Robin On Mon, Jun 30, 2008 at 6:08 PM, deneche abdelhakim <[EMAIL PROTECTED]> wrote: > In my case I used a SequenceFileOutputFormat, then I use the following code > to merge, sort and read the output: > > ... > import org.apache.hadoop.io.SequenceFile.Reader; > import org.apache.hadoop.io.SequenceFile.Sorter; > ... > > // list all files in the output path (each reducer should genrate a > different file) > public static Path[] listOutputFiles(FileSystem fs, Path outpath) > throws IOException { > FileStatus[] status = fs.listStatus(outpath); > List<Path> outpaths = new ArrayList<Path>(); > for (FileStatus s : status) { > if (!s.isDir()) { > outpaths.add(s.getPath()); > } > } > > Path[] outfiles = new Path[outpaths.size()]; > outpaths.toArray(outfiles); > > return outfiles; > } > > // merge, sort and read all the output files. > // the output keys are used to sort the output > // in my case the output key is a LongWritable, and the output value is a > FloatWritable > // (wich is converted to a Float), it should not be too difficult to use > other types > // the List evaluations will contain all the output values ordered by > their respective keys > public static void importEvaluations(FileSystem fs, JobConf conf, > Path outpath, List<Float> evaluations) throws IOException { > Sorter sorter = new Sorter(fs, LongWritable.class, FloatWritable.class, > conf); > > // merge and sort the outputs > Path[] outfiles = listOutputFiles(fs, outpath); > Path output = new Path(outpath, "output.sorted"); > sorter.merge(outfiles, output); > > // import the evaluations > LongWritable key = new LongWritable(); > FloatWritable value = new FloatWritable(); > Reader reader = new Reader(fs, output, conf); > > while (reader.next(key, value)) { > evaluations.add(value.get()); > } > > reader.close(); > } > > Hope this helps :P > > --- En date de : Lun 30.6.08, Grant Ingersoll <[EMAIL PROTECTED]> a > écrit : > De: Grant Ingersoll <[EMAIL PROTECTED]> > Objet: Re: About the Bayes TrainerDriver > À: [email protected] > Date: Lundi 30 Juin 2008, 13h52 > > I imagine there is a way to combine them or I am just doing something > stupid. Won't be the last time for that :-) > > -Grant > > On Jun 29, 2008, at 11:57 PM, Robin Anil wrote: > > > Hi, > > I am also stuck in the same place. The Trainer with multiple > > reducers > > generated as many number of part files as there are reducers. I have > > many > > questions in my mind. > > In what way are these part files generated? > > Is there a definite pattern? > > Inorder to read them should I run another MR job OR read through the > > files > > one by one? > > > > Any thoughts > > > > Robin > > > > _____________________________________________________________________________ > Envoyez avec Yahoo! Mail. Une boite mail plus intelligente > http://mail.yahoo.fr >
