My ops guys actually zipped them and who knows what version they used. I will try rezipping with the gzip on my Linux box and see if that fixes the problem. Then I'll try setting the native.lib property to false. Will keep you posted.
Thanks to everybody who has responded, Jeff -----Original Message----- From: Arun C Murthy [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 26, 2008 3:47 PM To: [email protected] Subject: Re: Decompression Blues Jeff, On Feb 26, 2008, at 12:58 PM, Jeff Eastman wrote: > I'm processing a number of .gz compressed Apache and other logs using > Hadoop 0.15.2 and encountering fatal decompression errors such as: > > How did you compress your input files? Could you share details on the version of your gzip and other tools? Try setting "hadoop.native.lib" property to 'false' via NativeCodeLoader.setLoadNativeLibraries for you job and see how it works... Arun > > 08/02/26 12:09:12 INFO mapred.JobClient: Task Id : > task_200802171116_0001_m_000005_0, Status : FAILED > > java.lang.InternalError > > at > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.init(Native > Method) > > at > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init> > (ZlibDecompres > sor.java:111) > > at > org.apache.hadoop.io.compress.GzipCodec.createDecompressor > (GzipCodec.jav > a:188) > > at > org.apache.hadoop.io.compress.GzipCodec.createInputStream > (GzipCodec.java > :170) > > at > org.apache.hadoop.mapred.LineRecordReader.<init> > (LineRecordReader.java:7 > 5) > > at > org.apache.hadoop.mapred.TextInputFormat.getRecordReader > (TextInputFormat > .java:50) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:156) > > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787) > > > > I looked in Jira but did not find any issues. Is this pilot error? > Some > of the files work just fine. Is there a workaround besides > unzipping all > the files in the DFS? > > > > Jeff >
