A quick workaround to get moving is to rename the Hadoop Native code that is causing native decompression to be attempted. The native binaries are in the distribution at ./lib/native/Linux-amd64-64 etc. Rename your architecture, and non-native code will be used.
Thanks, Stu -----Original Message----- From: Jeff Eastman <[EMAIL PROTECTED]> Sent: Tuesday, February 26, 2008 3:58pm To: [email protected] Subject: Decompression Blues I'm processing a number of .gz compressed Apache and other logs using Hadoop 0.15.2 and encountering fatal decompression errors such as: 08/02/26 12:09:12 INFO mapred.JobClient: Task Id : task_200802171116_0001_m_000005_0, Status : FAILED java.lang.InternalError at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.init(Native Method) at org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompres sor.java:111) at org.apache.hadoop.io.compress.GzipCodec.createDecompressor(GzipCodec.jav a:188) at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java :170) at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:7 5) at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat .java:50) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:156) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787) I looked in Jira but did not find any issues. Is this pilot error? Some of the files work just fine. Is there a workaround besides unzipping all the files in the DFS? Jeff
