error while using ArrayWritable and how to fix it?
Hello everyone,
I want to find out how to get an array as output in mapreduce.
I just simply modified the reduce method of wordcount example as shown below:
public static class IntSumReducer extends
ReducerText, IntWritable, Text, ArrayWritable {
private IntWritable[] iw = new IntWritable[2];
private ArrayWritable result = new ArrayWritable
(IntWritable.class,iw);
public void reduce(Text key, IterableIntWritable values,
Context context) throws IOException,
InterruptedException {
iw[0] = new IntWritable(); // initialize
iw[1] = new IntWritable(); // initialize
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
iw[0].set(sum);
iw[1].set(sum);
result.set(iw);
context.write(key, result);
}
}
Map outputs Text ,ArrayWritable.
Reduce takes Text,IterableIntWritable as inputs.
The moment I'm trying to use the ArrayWritable in reduce to store the results
I get the following error :
11/03/08 17:03:07 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
11/03/08 17:03:07 WARN mapred.JobClient: No job jar file set. User classes
may not be found. See JobConf(Class) or JobConf#setJar(String).
11/03/08 17:03:07 INFO input.FileInputFormat: Total input paths to process : 1
11/03/08 17:03:07 INFO mapred.JobClient: Running job: job_local_0001
11/03/08 17:03:07 INFO input.FileInputFormat: Total input paths to process : 1
11/03/08 17:03:07 INFO mapred.MapTask: io.sort.mb = 100
11/03/08 17:03:07 INFO mapred.MapTask: data buffer = 79691776/99614720
11/03/08 17:03:07 INFO mapred.MapTask: record buffer = 262144/327680
11/03/08 17:03:08 INFO mapred.MapTask: Starting flush of map output
11/03/08 17:03:08 WARN mapred.LocalJobRunner: job_local_0001
java.io.IOException: wrong value class: class
org.apache.hadoop.io.ArrayWritable is not class
org.apache.hadoop.io.IntWritable
at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:167)
at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect
(Task.java:880)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write
(Task.java:1201)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write
(TaskInputOutputContext.java:80)
at org.apache.hadoop.examples.WordCount$IntSumReducer.reduce
(WordCount.java:59)
at org.apache.hadoop.examples.WordCount$IntSumReducer.reduce
(WordCount.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine
(Task.java:1222)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill
(MapTask.java:1265)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush
(MapTask.java:1129)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close
(MapTask.java:549)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run
(LocalJobRunner.java:177)
11/03/08 17:03:08 INFO mapred.JobClient: map 0% reduce 0%
11/03/08 17:03:08 INFO mapred.JobClient: Job complete: job_local_0001
11/03/08 17:03:08 INFO mapred.JobClient: Counters: 0
Re: error while using ArrayWritable
Thanks J-D , worked perfectly fine ! On Sat, Jan 2, 2010 at 1:16 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote: This is explained in the javadoc: http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/io/ArrayWritable.html J-D On Fri, Jan 1, 2010 at 11:29 PM, bharath vissapragada bhara...@students.iiit.ac.in wrote: Hi all , I am using ArrayWritable in my MR job . Map outputs Text ,ArrayWritable Reduce takes Text,IterableArrayWritable The moment Iam trying to use the ArrayWritable in reduce using the iterator ..I get the following error : 10/01/02 18:23:41 WARN mapred.LocalJobRunner: job_local_0001 java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:81) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:940) at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:880) at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:237) at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:233) at HashJoin.MR_hash$redu.reduce(MR_hash.java:132) at mtr.MyTableReduce.reduce(MyTableReduce.java:1) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.init() at java.lang.Class.getConstructor0(Class.java:2723) at java.lang.Class.getDeclaredConstructor(Class.java:2002) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:75) ... 10 more java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) at HashJoin.MR_hash.run(MR_hash.java:294) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at HashJoin.MR_hash.join(MR_hash.java:320) at HashJoin.MR_hash.main(MR_hash.java:338) Can Anyone tell me what the error is ??
error while using ArrayWritable
Hi all , I am using ArrayWritable in my MR job . Map outputs Text ,ArrayWritable Reduce takes Text,IterableArrayWritable The moment Iam trying to use the ArrayWritable in reduce using the iterator ..I get the following error : 10/01/02 18:23:41 WARN mapred.LocalJobRunner: job_local_0001 java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:81) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:940) at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:880) at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:237) at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:233) at HashJoin.MR_hash$redu.reduce(MR_hash.java:132) at mtr.MyTableReduce.reduce(MyTableReduce.java:1) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.init() at java.lang.Class.getConstructor0(Class.java:2723) at java.lang.Class.getDeclaredConstructor(Class.java:2002) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:75) ... 10 more java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) at HashJoin.MR_hash.run(MR_hash.java:294) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at HashJoin.MR_hash.join(MR_hash.java:320) at HashJoin.MR_hash.main(MR_hash.java:338) Can Anyone tell me what the error is ??
Re: error while using ArrayWritable
This is explained in the javadoc: http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/io/ArrayWritable.html J-D On Fri, Jan 1, 2010 at 11:29 PM, bharath vissapragada bhara...@students.iiit.ac.in wrote: Hi all , I am using ArrayWritable in my MR job . Map outputs Text ,ArrayWritable Reduce takes Text,IterableArrayWritable The moment Iam trying to use the ArrayWritable in reduce using the iterator ..I get the following error : 10/01/02 18:23:41 WARN mapred.LocalJobRunner: job_local_0001 java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.init() at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:81) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:940) at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:880) at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:237) at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:233) at HashJoin.MR_hash$redu.reduce(MR_hash.java:132) at mtr.MyTableReduce.reduce(MyTableReduce.java:1) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.init() at java.lang.Class.getConstructor0(Class.java:2723) at java.lang.Class.getDeclaredConstructor(Class.java:2002) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:75) ... 10 more java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) at HashJoin.MR_hash.run(MR_hash.java:294) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at HashJoin.MR_hash.join(MR_hash.java:320) at HashJoin.MR_hash.main(MR_hash.java:338) Can Anyone tell me what the error is ??
