I am running a hadoop job that combines daily results with results with
previous days. The reduce output is lzo compressed and growing daily in
size.
- DistributedLzoIndexer is used to index lzo files to provide
parallelism. When the size of the lzo files were small, everything went
well. As the size of .lzo files grow, the chance that one of the partition
does not complete increases. The exception I got for one such case is
listed at the end of the post.
- It's possible to sacrifice parallelism by having hadoop work on each
.lzo file without indexing. This worked well until the file size exceeded
30G when array indexing exception got thrown. Apparently the code processed
the file in chunks and stored the references to the chunk in an array. When
the number of chunks was greater than a certain number (around 256 was my
recollection), exception was thrown.
- My current work around is to increase the number of reducers to keep
the .lzo file size low.
I would like to get advices on how people handle large .lzo files. Any
pointers on the cause of the stack trace below and best way to resolve it
are greatly appreciated.
Task Logs: 'attempt_201002041203_0032_m_000134_3'
syslog logs
2010-02-13 19:59:51,839 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processNa
me=MAP, sessionId=
2010-02-13 19:59:52,045 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2010-02-13 19:59:52,094 INFO org.apache.hadoop.mapred.MapTask: data buffer =
79691776/99614720
2010-02-13 19:59:52,094 INFO org.apache.hadoop.mapred.MapTask: record buffer
= 262144/327680
2010-02-13 19:59:52,099 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader:
Loaded native gpl library
2010-02-13 19:59:52,111 INFO com.hadoop.compression.lzo.LzoCodec:
Successfully loaded & initialized native-lzo
library [hadoop-lzo rev fatal: Not a git repository (or any of the parent
directories): .git]
2010-02-13 19:59:52,144 INFO com.baz.facetmr.mapreduce.foo.FooBar:
InputFile=hdfs://ndbig03:9000/user/hadoop/f
oo/basr_old/part-r-00000.lzo
2010-02-13 19:59:52,144 INFO com.baz.facetmr.mapreduce.foo.FooBar:
maxBarCount =1000
2010-02-13 19:59:55,267 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: record full = true
2010-02-13 19:59:55,268 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0;
bufend = 18641936; bufvoid = 9961
4720
2010-02-13 19:59:55,268 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0;
kvend = 262144; length = 327680
2010-02-13 19:59:55,958 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new compressor
2010-02-13 19:59:56,730 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 0
2010-02-13 19:59:58,567 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: record full = true
2010-02-13 19:59:58,567 INFO org.apache.hadoop.mapred.MapTask: bufstart =
18641936; bufend = 37532762; bufvoid
= 99614720
2010-02-13 19:59:58,567 INFO org.apache.hadoop.mapred.MapTask: kvstart =
262144; kvend = 196607; length = 3276
80
2010-02-13 19:59:59,487 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 1
2010-02-13 20:00:01,382 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: record full = true
2010-02-13 20:00:01,382 INFO org.apache.hadoop.mapred.MapTask: bufstart =
37532762; bufend = 56161807; bufvoid
= 99614720
2010-02-13 20:00:01,382 INFO org.apache.hadoop.mapred.MapTask: kvstart =
196607; kvend = 131070; length = 3276
80
2010-02-13 20:00:02,282 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 2
2010-02-13 20:00:04,100 INFO org.apache.hadoop.mapred.MapTask: Spilling map
output: record full = true
2010-02-13 20:00:04,100 INFO org.apache.hadoop.mapred.MapTask: bufstart =
56161807; bufend = 74935116; bufvoid
= 99614720
2010-02-13 20:00:04,100 INFO org.apache.hadoop.mapred.MapTask: kvstart =
131070; kvend = 65533; length = 32768
0
2010-02-13 20:00:05,100 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 3
2010-02-13 20:00:06,177 FATAL org.apache.hadoop.mapred.TaskTracker: Error
running child : java.lang.InternalEr
ror: lzo1x_decompress_safe returned:
at
com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect(Native
Method)
at
com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:303)
at
com.hadoop.compression.lzo.LzopDecompressor.decompress(LzopDecompressor.java:104)
at
com.hadoop.compression.lzo.LzopInputStream.decompress(LzopInputStream.java:223)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at java.io.InputStream.read(InputStream.java:85)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:187)
at
com.hadoop.mapreduce.LzoLineRecordReader.nextKeyValue(LzoLineRecordReader.java:126)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
