Hi Steve, On Sun, Feb 14, 2010 at 12:11 PM, Steve Kuo <[email protected]> wrote: > I am running a hadoop job that combines daily results with results with > previous days. The reduce output is lzo compressed and growing daily in > size. > > > - DistributedLzoIndexer is used to index lzo files to provide > parallelism. When the size of the lzo files were small, everything went > well. As the size of .lzo files grow, the chance that one of the partition > does not complete increases. The exception I got for one such case is > listed at the end of the post.
I'm not sure here whether you mean that the DistributedLzoIndexer job is failing, or if the job on the resulting split file is faiing. Could you clarify? > - It's possible to sacrifice parallelism by having hadoop work on each > .lzo file without indexing. This worked well until the file size exceeded > 30G when array indexing exception got thrown. Apparently the code processed > the file in chunks and stored the references to the chunk in an array. When > the number of chunks was greater than a certain number (around 256 was my > recollection), exception was thrown. > - My current work around is to increase the number of reducers to keep > the .lzo file size low. > > I would like to get advices on how people handle large .lzo files. Any > pointers on the cause of the stack trace below and best way to resolve it > are greatly appreciated. > Is this reproducible every time? If so, is it always at the same point in the LZO file that it occurs? Would it be possible to download that lzo file to your local box and use lzop -d to see if it decompresses successfully? That way we can isolate whether it's a compression bug or decompression. Thanks -Todd
