Thanks for your reply. I need to set up different types for output inside Map
and Reduce, those methods are inside main method: (InDegreeReduce and
InDegreeMap implements the interfaces)
conf.setMapOutputKeyClass(Text.class);
conf.setMapOutputValueClass(IntWritable.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(Text.class);
conf.setMapperClass(InDegreeMap.class);
// I have to denote the following line otherwise the exception come out
when running
// conf.setCombinerClass(InDegreeReduce.class);
conf.setReducerClass(InDegreeReduce.class);
I have to denote 'conf.setCombinerClass(...)' otherwise it reminds me errors as
below when running (compiled successfully)
------------------------------------------
08/08/18 17:45:52 INFO mapred.JobClient: Task Id :
task_200808161218_0077_m_000001_0, Status : FAILED
java.io.IOException: wrong value class: org.apache.hadoop.io.Text is not class
org.apache.hadoop.io.IntWritable
at
org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:998)
at
org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1083)
at DFS_InDegree$InDegreeReduce.reduce(DFS_InDegree.java:77)
at DFS_InDegree$InDegreeReduce.reduce(DFS_InDegree.java:52)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:876)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:782)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:694)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
--------------------------------------------
I think there are something happen when the combiner try to handle map and
reduce but don't know what exactly it is. thanks for your replay again.
Please let me know if you have any idea.
Best,
Keye
--- On Mon, 8/18/08, Ted Dunning <[EMAIL PROTECTED]> wrote:
> From: Ted Dunning <[EMAIL PROTECTED]>
> Subject: Re: OK to remove conf.setCombinerClass(myReduce.class)?
> To: [email protected], [EMAIL PROTECTED]
> Date: Monday, August 18, 2008, 8:57 PM
> This doesn't sound right.
>
> There are two reasons.
>
> First, the combiner is critical for good performance on
> word count
> applications.
>
> Secondly, having a combiner should not prevent having a
> different kind of
> collector. The combiner should just look like the reducer.
>
> Why do you think that having the combiner is causing your
> problem?
>
> On Mon, Aug 18, 2008 at 1:32 PM, Kunsheng Chen
> <[EMAIL PROTECTED]> wrote:
>
> > Hello, everyone.
> >
> > I am doing example WordCount and have to remove
> > conf.setCombinerClass(myReduce.class) in main method
> in order to have maper
> > and reduce run with different types of output
> collector.
> >
> > I am not sure whether it cause the problem for the
> perfomance. Please let
> > me know if it does. Any idea is appreciated.
> >
> >
> >
> >
>
>
>
>
> --
> ted
