If your Map and Reduce functions will be outputting different types of keys/values, then you will need to set the MapOutputKeyClass/MapOutputValueClass properties.
Your configuration will look like this instead: """ JobConf conf = new JobConf(test.class); conf.setMapOutputKeyClass(Text.class); conf.setMapOutputValueClass(Text.class); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(IntWritable.class); conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); """ Thanks, Stu -----Original Message----- From: slitz <[EMAIL PROTECTED]> Sent: Wednesday, February 27, 2008 9:08pm To: [email protected] Subject: Different output classes from map and reducer Hello, I'm experimenting with hadoop a few days now, but i'm stuck trying to output different classes from map and reduce methods. I have something like: class test { public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text> { public void map(LongWritable key, Text value, OutputCollector<Text, Text> output, Reporter reporter) throws IOException { (...) output.collect(new Text(...), new Text(...)); } } public static class Reduce extends MapReduceBase implements Reducer<Text, Text, Text, IntWritable> { public void reduce(Text key, Iterator<Text> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { (...) output.collect(key, new IntWritable(...)); } } } the relevant part of my conf goes like: JobConf conf = new JobConf(test.class); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(Text.class); conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); i keep getting this error: 08/02/28 01:52:47 INFO mapred.JobClient: map 50% reduce 0% 08/02/28 01:52:51 INFO mapred.JobClient: Task Id : task_200802261545_0032_m_000001_0, Status : FAILED java.io.IOException: wrong value class: org.apache.hadoop.io.IntWritable is not class org.apache.hadoop.io.Text at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:938) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$1.collect(MapTask.java :414) at org.myorg.WordCount$Reduce.reduce(WordCount.java:64) at org.myorg.WordCount$Reduce.reduce(WordCount.java:49) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill( MapTask.java:439) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk( MapTask.java:418) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:604) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:193) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1804) if in the conf i switch this -> conf.setOutputValueClass(Text.class); (map's output value type for this -> conf.setOutputValueClass(IntWritable.class); (reduce's output value type) then i get this: 08/02/28 02:05:08 INFO mapred.JobClient: map 50% reduce 0% 08/02/28 02:05:12 INFO mapred.JobClient: Task Id : task_200802261545_0033_m_000001_0, Status : FAILED java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.Text at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java :336) at org.myorg.WordCount$Map.map(WordCount.java:43) at org.myorg.WordCount$Map.map(WordCount.java:16) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1804) i'm just trying to modify slightly the wordcount example to fit my needs but i keep getting this kind of errors. Can somebody please point me the right direction? Thank you slitz
