Hello,
If my understanding is correct, the combiner will read in values for
a given key, process it, output it and then **all** values for a key
are given to the reducer.
Then it ought to be possible for the combiner to be of the form
public static class ClosestCenterCB extends MapReduceBase implements
Reducer<IntWritable, Text, IntWritable, BytesWritable>{
public void reduce(IntWritable key, Iterator<Text> values,
OutputCollector<IntWritable, BytesWritable> output, Reporter reporter)
{...}
}
and the reducer:
public static class ClosestCenterMR extends MapReduceBase implements
Mapper<LongWritable, Text, IntWritable, Text>, Reducer<IntWritable,
BytesWritable, IntWritable, Text>{
public void reduce(IntWritable key, Iterator<BytesWritable> values,
OutputCollector<IntWritable, Text> output, Reporter reporter) throws
IOException { ..}
}
However, when I set up the jobconf
theJob.setOutputKeyClass(IntWritable.class);
theJob.setOutputValueClass(Text.class);
theJob.setReducerClass(ClosestCenterMR.class);
theJob.setCombinerClass(ClosestCenterCB.class);
The outputvalue is TextClass and so I get the following error:
ava.io.IOException: wrong value class: class
org.apache.hadoop.io.BytesWritable is not class
org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:144)
at org.apache.hadoop.mapred.Task
$CombineOutputCollector.collect(Task.java:626)
at org.saptarshiguha.clusters.ClusterCenter
$ClosestCenterCB.reduce(Unknown Source)
at org.saptarshiguha.clusters.ClusterCenter
$ClosestCenterCB.reduce(Unknown Source)
at org.apache.hadoop.mapred.MapTask
$MapOutputBuffer.combineAndSpill(MapTask.java:904)
at org.apache.hadoop.mapred.MapTask
$MapOutputBuffer.sortAndSpill(MapTask.java:785)
at org.apache.hadoop.mapred.MapTask
$MapOutputBuffer.flush(MapTask.java:698)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:228)
at org.apache.hadoop.mapred.LocalJobRunner
$Job.run(LocalJobRunner.java:157)
08/11/16 16:48:18 INFO mapred.LocalJobRunner: file:/tmp/input/
sample.data:0+308
Saptarshi Guha | [EMAIL PROTECTED] | http://www.stat.purdue.edu/~sguha
This is your fortune.
