Hi Alex, Unfortunately I wasn't able to reproduce, and the data Dmitriy is working with is sensitive. Do you have some data you could upload (or send me off list) that exhibits the issue? -Todd
On Tue, Apr 6, 2010 at 9:50 AM, Alex Roetter <[email protected]> wrote: > > Todd Lipcon <t...@...> writes: > > > > > Hey Dmitriy, > > > > This is very interesting (and worrisome in a way!) I'll try to take a look > > this afternoon. > > > > -Todd > > > > Hi Todd, > > I wanted to see if you made any progress on this front. I'm seeing a very > similar error, trying to run a MR (Hadoop 0.20.1) over a bunch of > LZOP compressed / indexed files (using Kevin Weil's package), and I have one > map task that always fails in what looks like the same place as described in > the previous post. I haven't yet done the experimentation mentioned above > (isolating the input file corresponding to the failed map task, decompressing > it / recompressing it, testing it out operating directly on local disk > instead of HDFS, etc). > > However, since I am crashing in exactly the same place it seems likely this > is related, and thought I'd check on your work in the meantime. > > FYI, my stack track is below: > > 2010-04-05 18:15:16,895 FATAL org.apache.hadoop.mapred.TaskTracker: Error > running child : java.lang.InternalError: lzo1x_decompress_safe returned: > at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect > (Native Method) > at com.hadoop.compression.lzo.LzoDecompressor.decompress > (LzoDecompressor.java:303) > at > com.hadoop.compression.lzo.LzopDecompressor.decompress > (LzopDecompressor.java:104) > at com.hadoop.compression.lzo.LzopInputStream.decompress > (LzopInputStream.java:223) > at > org.apache.hadoop.io.compress.DecompressorStream.read > (DecompressorStream.java:74) > at java.io.InputStream.read(InputStream.java:85) > at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134) > at org.apache.hadoop.util.LineReader.readLine(LineReader.java:187) > at > com.hadoop.mapreduce.LzoLineRecordReader.nextKeyValue > (LzoLineRecordReader.java:126) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue > (MapTask.java:423) > at > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > > Any update much appreciated, > Alex > > > > > -- Todd Lipcon Software Engineer, Cloudera
