What you need to do is snag access to the filename in the configure method of the mapper.
Then instead of outputting just the word as the key, output a pair containing the word and the file name as the key. Everything downstream should remain the same. On Sun, Oct 5, 2008 at 11:26 AM, Alex Loddengaard <[EMAIL PROTECTED] > wrote: > Hi Srilatha, > > You could do the following: > > Map steps can only collect key, value pairs; they can't collect three > things. Instead of collecting (word, 1) in the mapper, you could collect > (word, filename). Then, in the reduce step, you could output (filename, > "word|count"), where "word|count" is Text, and "word" and "count" are both > variables. This solution would require changing the value type from > IntWritable to Text. > > As for getting the file name, read more here: > > < > > http://www.nabble.com/Processing-multiple-files---need-to-identify-in-map-td15841235.html > > > > Alex > > On Sun, Oct 5, 2008 at 10:12 AM, Latha <[EMAIL PROTECTED]> wrote: > > > Greetings! > > > > Hi, Am trying to modify the WordCount.java mentioned at Example: > WordCount > > v1.0< > > > http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Example%3A+WordCount+v1.0 > > >at > > http://hadoop.apache.org/core/docs/current/mapred_tutorial.html > > Would like to have output the following way, > > > > FileOne word1 itsCount > > FileOne word2 itsCount > > ..(and so on) > > FileTwo word1 itsCount > > FileTwo wordx its Count > > .. > > FileThree word1 its Count > > .. > > > > Am trying to do following changes to the code of WordCount.java > > > > 1) private Text filename = new Text(); // Added this to Map class .Not > > sure if I would have access to filename here. > > 2) (line 18)OutputCollector<Text, Text, IntWritable> output // Changed > > the > > argument in the map() function to have another Text field. > > 3) (line 23) output.collect(filename, word , one); // Trying to change > the > > output format as 'filename word count' > > > > Am not sure what other changes are to be affected to achieve the required > > output. filename is not available to the map method. > > My requirement is to go through all the data available in hdfs and > prepare > > an index file with < filename word count> format. > > Could you please throw light on how I can achieve this. > > > > Thankyou > > Srilatha > > > -- ted
