Jeff, if I were to debug this I would write a M/R job that generates some random data and uses Zlib for output compression. Then you can try to read the output of the job again and see if the issue is still there. We use Hadoop native libraries for compression all the time and I have never seen this exception before. One thing though, we usually use compressed SequenceFiles, but from your stack trace I can see that you are working with a compressed text file. In your tests, you might want to try both output formats and see if you encounter the same problem.
RB On 2/27/08, Jeff Eastman <[EMAIL PROTECTED]> wrote: > > I unzipped and rezipped all the files using gzip 1.3.3 and uploaded the > files again. I got the same exceptions. > > I set the hadoop.native.lib property to false and bounced the cloud, > then ran my job. I still get the same exceptions. > > Any more suggestions? > > > Jeff > > -----Original Message----- > From: Arun C Murthy [mailto:[EMAIL PROTECTED] > Sent: Tuesday, February 26, 2008 3:47 PM > To: [email protected] > Subject: Re: Decompression Blues > > > Jeff, > > On Feb 26, 2008, at 12:58 PM, Jeff Eastman wrote: > > > I'm processing a number of .gz compressed Apache and other logs using > > Hadoop 0.15.2 and encountering fatal decompression errors such as: > > > > > > How did you compress your input files? Could you share details on the > version of your gzip and other tools? > > Try setting "hadoop.native.lib" property to 'false' via > NativeCodeLoader.setLoadNativeLibraries for you job and see how it > works... > > Arun > > > > > 08/02/26 12:09:12 INFO mapred.JobClient: Task Id : > > task_200802171116_0001_m_000005_0, Status : FAILED > > > > java.lang.InternalError > > > > at > > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.init(Native > > Method) > > > > at > > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init> > > (ZlibDecompres > > sor.java:111) > > > > at > > org.apache.hadoop.io.compress.GzipCodec.createDecompressor > > (GzipCodec.jav > > a:188) > > > > at > > org.apache.hadoop.io.compress.GzipCodec.createInputStream > > (GzipCodec.java > > :170) > > > > at > > org.apache.hadoop.mapred.LineRecordReader.<init> > > (LineRecordReader.java:7 > > 5) > > > > at > > org.apache.hadoop.mapred.TextInputFormat.getRecordReader > > (TextInputFormat > > .java:50) > > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:156) > > > > at > > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787) > > > > > > > > I looked in Jira but did not find any issues. Is this pilot error? > > Some > > of the files work just fine. Is there a workaround besides > > unzipping all > > the files in the DFS? > > > > > > > > Jeff > > > > -- ------------------------------- Riccardo Boscolo, PhD V.P. of Core Technology Netseer Inc. 11943 Montana Ave, Suite 200 Los Angeles, CA 90049 T: 310-597-4482 F: 310-597-4489 Email: [EMAIL PROTECTED] -------------------------------
