I'm processing a number of .gz compressed Apache and other logs using
Hadoop 0.15.2 and encountering fatal decompression errors such as:
08/02/26 12:09:12 INFO mapred.JobClient: Task Id :
task_200802171116_0001_m_000005_0, Status : FAILED
java.lang.InternalError
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.init(Native Method)
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompres
sor.java:111)
at
org.apache.hadoop.io.compress.GzipCodec.createDecompressor(GzipCodec.jav
a:188)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java
:170)
at
org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:7
5)
at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat
.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:156)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787)
I looked in Jira but did not find any issues. Is this pilot error? Some
of the files work just fine. Is there a workaround besides unzipping all
the files in the DFS?
Jeff
