I ran the job again, but with less other processes running on the same machine,
so with more physical memory available to HBase. This was to see whether there
was a point where it would stop allocating more buffers. When I do this, after
many hours, one of the RSes crashed with a OOME. See here:
2011-01-04 11:32:01,332 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
serverName=w5r1.inrdb.ripe.net,60020,1294091507228, load=(requests=6246,
regions=258, usedHeap=1790, maxHeap=16000): Uncaught exception in service
thread regionserver60020.compactor
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:633)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:98)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at com.hadoop.compression.lzo.LzoCompressor.init(LzoCompressor.java:248)
at
com.hadoop.compression.lzo.LzoCompressor.reinit(LzoCompressor.java:207)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:105)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
at
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:200)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.getCompressingStream(HFile.java:397)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.newBlock(HFile.java:383)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.checkBlockBoundary(HFile.java:354)
at org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:536)
at org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:501)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:836)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:931)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:764)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:709)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:81)
2011-01-04 11:32:01,369 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=258, stores=516, storefiles=186, storefileIndexSize=179,
memstoreSize=2125, compactionQueueSize=2, usedHeap=1797, maxHeap=16000,
blockCacheSize=55051488, blockCacheFree=6655834912, blockCacheCount=0,
blockCacheHitCount=0, blockCacheMissCount=2397107, blockCacheEvictedCount=0,
blockCacheHitRatio=0, blockCacheHitCachingRatio=0
I am guessing the OS won't allocate any more memory to the process. As you can
see, the used heap is nowhere near the max heap.
Also, this happens from the compaction, it seems. I had not considered those as
a suspect yet. I could try running with a larger compaction threshold and
blocking store files. Since this is a write only load, it should not be a
problem. In our normal operation, compactions and splits are quite common,
though, because we do read-modify-write cycles a lot. Anyone else doing update
heavy work with LZO?
Cheers,
Friso
On 4 jan 2011, at 01:54, Todd Lipcon wrote:
> Fishy. Are your cells particularly large? Or have you tuned the HFile block
> size at all?
>
> -Todd
>
> On Mon, Jan 3, 2011 at 2:15 PM, Friso van Vollenhoven <
> [email protected]> wrote:
>
>> I tried it, but it doesn't seem to help. The RS processes grow to 30Gb in
>> minutes after the job started.
>>
>> Any ideas?
>>
>>
>> Friso
>>
>>
>>
>> On 3 jan 2011, at 19:18, Todd Lipcon wrote:
>>
>>> Hi Friso,
>>>
>>> Which OS are you running? Particularly, which version of glibc?
>>>
>>> Can you try running with the environment variable MALLOC_ARENA_MAX=1 set?
>>>
>>> Thanks
>>> -Todd
>>>
>>> On Mon, Jan 3, 2011 at 8:15 AM, Friso van Vollenhoven <
>>> [email protected]> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I seem to run into a problem that occurs when using LZO compression on a
>>>> heavy write only load. I am using 0.90 RC1 and, thus, the LZO compressor
>>>> code that supports the reinit() method (from Kevin Weil's github,
>> version
>>>> 0.4.8). There are some more Hadoop LZO incarnations, so I am pointing my
>>>> question to this list.
>>>>
>>>> It looks like the compressor uses direct byte buffers to store the
>> original
>>>> and compressed bytes in memory, so the native code can work with it
>> without
>>>> the JVM having to copy anything around. The direct buffers are possibly
>>>> reused after a reinit() call, but will often be newly created in the
>> init()
>>>> method, because the existing buffer can be the wrong size for reusing.
>> The
>>>> latter case will leave the previously used buffers by the compressor
>>>> instance eligible for garbage collection. I think the problem is that
>> this
>>>> collection never occurs (in time), because the GC does not consider it
>>>> necessary yet. The GC does not know about the native heap and based on
>> the
>>>> state of the JVM heap, there is no reason to finalize these objects yet.
>>>> However, direct byte buffers are only freed in the finalizer, so the
>> native
>>>> heap keeps growing. On write only loads, a full GC will rarely happen,
>>>> because the max heap will not grow far beyond the mem stores (no block
>> cache
>>>> is used). So what happens is that the machine starts using swap before
>> the
>>>> GC will ever clean up the direct byte buffers. I am guessing that
>> without
>>>> the reinit() support, the buffers were collected earlier because the
>>>> referring objects would also be collected every now and then or things
>> would
>>>> perhaps just never promote to an older generation.
>>>>
>>>> When I do a pmap on a running RS after it has grown to some 40Gb
>> resident
>>>> size (with a 16Gb heap), it will show a lot of near 64M anon blocks
>>>> (presumably native heap). I show this before with the 0.4.6 version of
>>>> Hadoop LZO, but that was under normal load. After that I went back to a
>>>> HBase version that does not require the reinit(). Now I am on 0.90 with
>> the
>>>> new LZO, but never did a heavy load like this one with that, until
>> now...
>>>>
>>>> Can anyone with a better understanding of the LZO code confirm that the
>>>> above could be the case? If so, would it be possible to change the LZO
>>>> compressor (and decompressor) to use maybe just one fixed size buffer
>> (they
>>>> all appear near 64M anyway) or possibly reuse an existing buffer also
>> when
>>>> it is not the exact required size but just large enough to make do?
>> Having
>>>> short lived direct byte buffers is apparently a discouraged practice. If
>>>> anyone can provide some pointers on what to look out for, I could invest
>>>> some time in creating a patch.
>>>>
>>>>
>>>> Thanks,
>>>> Friso
>>>>
>>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
