[
https://issues.apache.org/jira/browse/MAPREDUCE-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079969#comment-13079969
]
Devaraj K commented on MAPREDUCE-2564:
--------------------------------------
As per the code, we cannot use the WritableComparator directly because the
constructors are made as protected. If there is no comparator for the output
key class, it uses this way to create instance of it and it initializes the
buffer and keys.
{code:xml}
public static synchronized WritableComparator get(Class<? extends
WritableComparable> c) {
WritableComparator comparator = comparators.get(c);
if (comparator == null)
comparator = new WritableComparator(c, true);
return comparator;
}
{code}
When we are writing custom comparator and if we don’t want to use the existing
compare() method we can use the below constructor from the subclass
{code:xml}
protected WritableComparator(Class<? extends WritableComparable> keyClass) {
this(keyClass, false);
}
{code}
Otherwise we can use this constructor
{code:xml}
protected WritableComparator(Class<? extends WritableComparable> keyClass,
boolean createInstances)
{code}
Can you elaborate the usage scenario?
> NullPointerException in WritableComparator
> ------------------------------------------
>
> Key: MAPREDUCE-2564
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2564
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.203.0
> Environment: java version "1.6.0_23"
> Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
> hadoop-0.20.203.0rc1
> Reporter: Joseph Shraibman
> Priority: Blocker
>
> java.lang.NullPointerException
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1110)
> at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1398)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1297)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
> It is easy to see why this is happening. The WritableComparator is created
> in JobConf line 776:
> {code:title=JobConf.java}
> return
> WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class));
> }
> {code}
> which calls
> {code:title=WritableComparator.java|borderStyle=solid}
> protected WritableComparator(Class<? extends WritableComparable> keyClass) {
> this(keyClass, false);
> }
> protected WritableComparator(Class<? extends WritableComparable> keyClass,
> boolean createInstances) {
> this.keyClass = keyClass;
> if (createInstances) {
> key1 = newKey();
> key2 = newKey();
> buffer = new DataInputBuffer();
> } else {
> key1 = key2 = null;
> buffer = null;
> }
> }
> {code}
> key1, key2, and buffer end up being null. When compare() is called the NPE is
> thrown because buffer is null
> {code:title=WritableComparator.java|borderStyle=solid}
> public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
> try {
> buffer.reset(b1, s1, l1); // parse key1
> key1.readFields(buffer);
>
> buffer.reset(b2, s2, l2); // parse key2
> key2.readFields(buffer);
>
> } catch (IOException e) {
> throw new RuntimeException(e);
> }
>
> return compare(key1, key2); // compare them
> }
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
