task fails with NPE when GzipCodec is used for
mapred.map.output.compression.codec and native libary is not present
--------------------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-857
URL: https://issues.apache.org/jira/browse/MAPREDUCE-857
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Reporter: Karam Singh
Fix For: 0.21.0
Ran a job with
mapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec.
Whenmaps of job completes they with following NPE -:
tasklog -:
2009-08-12 13:48:13,423 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 256
2009-08-12 13:48:13,611 INFO org.apache.hadoop.mapred.MapTask: data buffer =
204010944/214748368
2009-08-12 13:48:13,611 INFO org.apache.hadoop.mapred.MapTask: record buffer =
3187670/3355443
2009-08-12 13:49:45,473 INFO org.apache.hadoop.mapred.MapTask: Starting flush
of map output
2009-08-12 13:49:45,544 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to
load native-hadoop library for your platform... using builtin-java classes
where applicable
2009-08-12 13:49:45,545 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new compressor
2009-08-12 13:49:45,546 WARN org.apache.hadoop.mapred.Child: Error running
child : java.lang.NullPointerException
at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:105)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1248)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1146)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:528)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:604)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:318)
at org.apache.hadoop.mapred.Child.main(Child.java:162)
Line 105 of IFile.java contains followings line in trunk code on which error
was seen -:
Line 104: this.compressor = CodecPool.getCompressor(codec);
Line: this.compressor.reset();
If native is available job runs successfully without any failures
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
