Hello,
I'm attempting to use a SortedMapWritable with a LongWritable as the
key and a custom implementation of org.apache.hadoop.io.Writable as
the value. I notice that my program works fine when I use another
primitive wrapper (e.g. Text) as the value, but fails with the
following exception when I use my custom Writable instance:
2008-09-08 23:25:02,072 INFO org.apache.hadoop.mapred.ReduceTask:
Initiating in-memory merge with 1 segments...
2008-09-08 23:25:02,077 INFO org.apache.hadoop.mapred.Merger: Merging
1 sorted segments
2008-09-08 23:25:02,077 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 1 segments left of total size: 5492 bytes
2008-09-08 23:25:02,099 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_200809082247_0005_r_000000_0 Merge of the inmemory files threw
a
n exception: java.io.IOException: Intermedate merge failed
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2133)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2064)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:80)
at
org.apache.hadoop.io.SortedMapWritable.readFields(SortedMapWritable.java:179)
...
I noticed that the AbstractMapWritable class has a protected
"addToMap(Class clazz)" method. Do I somehow need to let my
SortedMapWritable instance know about my custom Writable value? I've
properly implemented the custom Writable object (it just contains a
few primitives, like longs and ints).
Any insight is appreciated.
Thanks,
Ryan
