In this case you probably want two different classes. You can have the base
Reducer class that adds in the line count, and then subclass it for the
combiner, that sets a flag to not output the line numbers.
--Bobby
On 6/9/11 12:57 PM, "Shi Yu" <sh...@uchicago.edu> wrote:
Hi,
Thanks for the reply. The line count in new API works fine now, it was a
bug in my code. In new API,
Iterator is changed to Iterable,
but I didn't pay attention to that and was still using Iterator and hasNext(),
Next() method. Surprisingly, the wrong code still ran and got output, but the
line number count did not work and I think it was null value. After fixing that
Iterable mistake, the code works fine.
The remaining problem is when combiner and reducer are both implemented, the
output is like
00000 00000 value1
00001 00000 value2
00002 00000 value3
00003 00001 value4
00004 00001 value5
The first column are counts from reducer, the second column are counts from
combiner. I want to avoid the line counter in combiner, so my plan is to create
another class which is almost the same as Reducer, but without the line count.
I think it is doable to set Combiner and Reducer to different classes in
jobconf, but I haven't tried it yet.
Best,
Shi
On 6/9/2011 8:49 AM, Robert Evans wrote:
> What exactly is linecount being output as in the new APIs?
>
> --Bobby
>
> On 6/7/11 11:21 AM, "Shi Yu"<sh...@uchicago.edu> wrote:
>
> Hi,
>
> I am wondering is there any built-in function to automatically add a
> self-increment line number in reducer output (like the relation DB
> auto-key).
>
> I have this problem because in 0.19.2 API, I used a variable linecount
> increasing in the reducer like:
>
> public static class Reduce extends MapReduceBase implements
> Reducer<Text, IntWritable, Text,IntWritable>{
> private long linecount = 0;
>
> public void reduce(Text key, Iterator<IntWritable> values,
> OutputCollector<Text, IntWritable> output, Reporter reporter) throws
> IOException {
>
> //.....some code here
> linecount ++;
> output.collect(new Text(Long.toString(linecount)), var);
>
> }
>
> }
>
>
> However, I found that this is not working in 0.20.2 API, if I write the
> code like:
>
> public static class Reduce extends
> org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable>{
> private long linecount = 0;
>
> public void reduce (Text key, Iterator<IntWritable> values,
> org.apache.hadoop.mapreduce.Reducer.Context context) throws IOException,
> InterruptedException {
>
> //some code here
> linecount ++;
> context.write(new Text(Long.toString(linecount)),var);
> }
> }
>
> but it seems not working anymore.
>
>
> I would also like to know if there are combiner and reducer implemented,
> how to avoid that line number being written twice (cause I only want it
> in reducer, not in combiner). Thanks!
>
>
> Shi
>
>
>
>
