[
https://issues.apache.org/jira/browse/MAPREDUCE-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079994#comment-13079994
]
Joseph Shraibman commented on MAPREDUCE-2564:
---------------------------------------------
Hmm. This was fixed between 0.20.2 and 0.20.203. It seems I reported this
against 203. Is it possible I used 0.20.2 and looked at the source of 0.20.2
while thinking it was 203? It is hard to remember what I was doing on that day
two months ago.
> NullPointerException in WritableComparator
> ------------------------------------------
>
> Key: MAPREDUCE-2564
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.203.0
> Environment: java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
> hadoop-0.20.203.0rc1
> Reporter: Joseph Shraibman
> Priority: Blocker
> Fix For: 0.20.203.0
>
>
> java.lang.NullPointerException
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
> at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
> It is easy to see why this is happening. The WritableComparator is created
> in JobConf line 776:
> {code:title=JobConf.java}
> return
> WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
> }
> {code}
> which calls
> {code:title=WritableComparator.java|borderStyle=solid}
> protected WritableComparator(Class<? extends WritableComparable> keyClass) {
> this(keyClass, false);
> }
> protected WritableComparator(Class<? extends WritableComparable> keyClass,
> boolean createInstances) {
> this.keyClass = keyClass;
> if (createInstances) {
> key1 = newKey();
> key2 = newKey();
> buffer = new DataInputBuffer();
> } else {
> key1 = key2 = null;
> buffer = null;
> }
> }
> {code}
> key1, key2, and buffer end up being null. When compare() is called the NPE is
> thrown because buffer is null
> {code:title=WritableComparator.java|borderStyle=solid}
> public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
> try {
> buffer.reset(b1, s1, l1); // parse key1
> key1.readFields(buffer);
>
> buffer.reset(b2, s2, l2); // parse key2
> key2.readFields(buffer);
>
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
>
> return compare(key1, key2); // compare them
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
