We have 16 m1.xlarge ec2 machines as region servers, running cdh3u2,
hosting about 17k regions. Each region server has 10GB of heap, and in
normal operating levels I have never seen our used heap go above 5-8GB.
Yesterday we were running a job to populate a new table, and this resulted
in a cascading OOM failure which ended with all region servers being down.
The failure on each node went something like this (region A is the same
region across all servers, getting passed along as each dies):
1. RS inherits region A.
2. RS tries to flush region A, but the region has "too many store
files". RS delays flush and instead runs a compaction
3. 1 minute pause in the logs (could have been a GC, the logs pretty
steadily were coming in every 1-2 seconds), results in lost connection to ZK
4. RS reconnects to ZK and blocks updates on region A, due to memstore
too big (129.8m is > 128m blocking size)
5. Another 30 second pause (another GC?)
6. Lost connection to server from master
7. 1-2 minutes later, aborts the compaction and throws OutOfMemoryError:
Java heap space. The exception comes from the compaction (pasted below).
That pattern repeats on all of the region servers, every 5-8 minutes until
all are down. Should there be some safeguards on a compaction causing a
region server to go OOM? The region appears to only be around 425mb in
size.
---
(This exception comes from the first regionserver to go down. The others
were very similar, with the same stacktrace.)
12/04/10 08:04:57 INFO regionserver.HRegion: aborted compaction on region
analytics-search-a2,\x00\x00^\xF4\x00\x0A0,1334056032860.ab55c22574a9cddec8a3e73fd99be57d.
after 14mins, 34sec
12/04/10 08:04:57 FATAL regionserver.HRegionServer: ABORTING region server
serverName=XXXXXXXX,60020,1326728856867, load=(requests=65, regions=1082,
usedHeap=10226, maxHeap=10231): Uncaught exception in service thread
regionserver60020.compactor
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.io.compress.DecompressorStream.<init>(DecompressorStream.java:43)
at
org.apache.hadoop.io.compress.BlockDecompressorStream.<init>(BlockDecompressorStream.java:45)
at
com.hadoop.compression.lzo.LzoCodec.createInputStream(LzoCodec.java:173)
at
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.createDecompressionStream(Compression.java:206)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1087)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1280)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:87)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:82)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:262)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:326)
at
org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:943)
at
org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:743)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:808)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:748)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:81)
