Jeff,
On Feb 26, 2008, at 12:58 PM, Jeff Eastman wrote:
I'm processing a number of .gz compressed Apache and other logs using
Hadoop 0.15.2 and encountering fatal decompression errors such as:
How did you compress your input files? Could you share details on the
version of your gzip and other tools?
Try setting "hadoop.native.lib" property to 'false' via
NativeCodeLoader.setLoadNativeLibraries for you job and see how it
works...
Arun
08/02/26 12:09:12 INFO mapred.JobClient: Task Id :
task_200802171116_0001_m_000005_0, Status : FAILED
java.lang.InternalError
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.init(Native
Method)
at
org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>
(ZlibDecompres
sor.java:111)
at
org.apache.hadoop.io.compress.GzipCodec.createDecompressor
(GzipCodec.jav
a:188)
at
org.apache.hadoop.io.compress.GzipCodec.createInputStream
(GzipCodec.java
:170)
at
org.apache.hadoop.mapred.LineRecordReader.<init>
(LineRecordReader.java:7
5)
at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader
(TextInputFormat
.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:156)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787)
I looked in Jira but did not find any issues. Is this pilot error?
Some
of the files work just fine. Is there a workaround besides
unzipping all
the files in the DFS?
Jeff
