Somewhere like imageshack :) http://img221.imageshack.us/img221/761/screenshot20110515at537p.png
-Jack On Sun, May 15, 2011 at 7:09 PM, Ted Yu <[email protected]> wrote: > Jack: > Your attachment got filtered. > Can you upload somewhere ? > > Thanks > > On Sun, May 15, 2011 at 5:37 PM, Jack Levin <[email protected]> wrote: > >> I've added occupancy: export HBASE_OPTS="$HBASE_OPTS -verbose:gc >> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails >> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError >> -Xloggc:$HBASE_HOME/logs/gc-hbase.log" >> >> But we had same issue. Whats strange is that the load on the RS would >> be around 1000 rpc calls, then we have hlog roll, and GC goes crazy, >> and kills all of the user CPU. I have another jstack: >> http://pastebin.com/4n3MvmrX from a different server. How do people >> deal with this? We have HEAP of 12G, block cace at 5G, and upper >> memstore at 1G. Is hlog roll that intensive? I though 1000 rpc calls >> per RS should be easy? The server is not at all busy pre-spike, you >> can tell by attached graph. >> >> -Jack >> >> On Sat, May 14, 2011 at 7:12 PM, Jack Levin <[email protected]> wrote: >> > Greetings all -- We had some issues with huge GC counts that drive CPU >> > to 100% and rendering RS basically dead, here is some data: >> > >> > We are normal here, about 1 ParNew per second. >> > >> > 67 2011-05-14T15:35 >> > 75 2011-05-14T15:36 >> > 70 2011-05-14T15:37 >> > 74 2011-05-14T15:38 >> > 73 2011-05-14T15:39 >> > 73 2011-05-14T15:40 >> > 69 2011-05-14T15:41 >> > 68 2011-05-14T15:42 >> > 76 2011-05-14T15:43 >> > 70 2011-05-14T15:44 >> > 68 2011-05-14T15:45 >> > 322 2011-05-14T15:46 >> > 562 2011-05-14T15:47 <------------ This is where its about 10 per >> second. >> > 567 2011-05-14T15:48 >> > 563 2011-05-14T15:49 >> > 562 2011-05-14T15:50 >> > 561 2011-05-14T15:51 >> > 513 2011-05-14T15:52 >> > >> > here is the jstack from the time of this: >> > >> > http://pastebin.com/RYdwH8Gp Unusually high BLOCKED threads. >> > >> > This is config for hbase-env: >> > >> > export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails >> > -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError >> > -Xloggc:$HBASE_HOME/logs/gc-hbase.log" >> > >> > Does any have any suggestions on how to manage this? >> > >> > -Jack >> > >> >
