If you have a high insert rate then maybe log rolling (which blocks inserts a little) makes it that the calls get queued enough (occupying heap) to make you enter a GC loop of death? Can you enable RPC logging and see if you can confirm that?
Thx, J-D On Sun, May 15, 2011 at 5:37 PM, Jack Levin <magn...@gmail.com> wrote: > I've added occupancy: export HBASE_OPTS="$HBASE_OPTS -verbose:gc > -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError > -Xloggc:$HBASE_HOME/logs/gc-hbase.log" > > But we had same issue. Whats strange is that the load on the RS would > be around 1000 rpc calls, then we have hlog roll, and GC goes crazy, > and kills all of the user CPU. I have another jstack: > http://pastebin.com/4n3MvmrX from a different server. How do people > deal with this? We have HEAP of 12G, block cace at 5G, and upper > memstore at 1G. Is hlog roll that intensive? I though 1000 rpc calls > per RS should be easy? The server is not at all busy pre-spike, you > can tell by attached graph. > > -Jack > > On Sat, May 14, 2011 at 7:12 PM, Jack Levin <magn...@gmail.com> wrote: >> Greetings all -- We had some issues with huge GC counts that drive CPU >> to 100% and rendering RS basically dead, here is some data: >> >> We are normal here, about 1 ParNew per second. >> >> 67 2011-05-14T15:35 >> 75 2011-05-14T15:36 >> 70 2011-05-14T15:37 >> 74 2011-05-14T15:38 >> 73 2011-05-14T15:39 >> 73 2011-05-14T15:40 >> 69 2011-05-14T15:41 >> 68 2011-05-14T15:42 >> 76 2011-05-14T15:43 >> 70 2011-05-14T15:44 >> 68 2011-05-14T15:45 >> 322 2011-05-14T15:46 >> 562 2011-05-14T15:47 <------------ This is where its about 10 per second. >> 567 2011-05-14T15:48 >> 563 2011-05-14T15:49 >> 562 2011-05-14T15:50 >> 561 2011-05-14T15:51 >> 513 2011-05-14T15:52 >> >> here is the jstack from the time of this: >> >> http://pastebin.com/RYdwH8Gp Unusually high BLOCKED threads. >> >> This is config for hbase-env: >> >> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails >> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError >> -Xloggc:$HBASE_HOME/logs/gc-hbase.log" >> >> Does any have any suggestions on how to manage this? >> >> -Jack >> >
