Todd Lipcon <t...@...> writes:
>
> Hey Dmitriy,
>
> This is very interesting (and worrisome in a way!) I'll try to take a look
> this afternoon.
>
> -Todd
>
Hi Todd,
I wanted to see if you made any progress on this front. I'm seeing a very
similar error, trying to run a MR (Hadoop 0.20.1) over a bunch of
LZOP compressed / indexed files (using Kevin Weil's package), and I have one
map task that always fails in what looks like the same place as described in
the previous post. I haven't yet done the experimentation mentioned above
(isolating the input file corresponding to the failed map task, decompressing
it / recompressing it, testing it out operating directly on local disk
instead of HDFS, etc).
However, since I am crashing in exactly the same place it seems likely this
is related, and thought I'd check on your work in the meantime.
FYI, my stack track is below:
2010-04-05 18:15:16,895 FATAL org.apache.hadoop.mapred.TaskTracker: Error
running child : java.lang.InternalError: lzo1x_decompress_safe returned:
at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect
(Native Method)
at com.hadoop.compression.lzo.LzoDecompressor.decompress
(LzoDecompressor.java:303)
at
com.hadoop.compression.lzo.LzopDecompressor.decompress
(LzopDecompressor.java:104)
at com.hadoop.compression.lzo.LzopInputStream.decompress
(LzopInputStream.java:223)
at
org.apache.hadoop.io.compress.DecompressorStream.read
(DecompressorStream.java:74)
at java.io.InputStream.read(InputStream.java:85)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:187)
at
com.hadoop.mapreduce.LzoLineRecordReader.nextKeyValue
(LzoLineRecordReader.java:126)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue
(MapTask.java:423)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Any update much appreciated,
Alex
