Jack: Your attachment got filtered. Can you upload somewhere ? Thanks
On Sun, May 15, 2011 at 5:37 PM, Jack Levin <[email protected]> wrote: > I've added occupancy: export HBASE_OPTS="$HBASE_OPTS -verbose:gc > -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError > -Xloggc:$HBASE_HOME/logs/gc-hbase.log" > > But we had same issue. Whats strange is that the load on the RS would > be around 1000 rpc calls, then we have hlog roll, and GC goes crazy, > and kills all of the user CPU. I have another jstack: > http://pastebin.com/4n3MvmrX from a different server. How do people > deal with this? We have HEAP of 12G, block cace at 5G, and upper > memstore at 1G. Is hlog roll that intensive? I though 1000 rpc calls > per RS should be easy? The server is not at all busy pre-spike, you > can tell by attached graph. > > -Jack > > On Sat, May 14, 2011 at 7:12 PM, Jack Levin <[email protected]> wrote: > > Greetings all -- We had some issues with huge GC counts that drive CPU > > to 100% and rendering RS basically dead, here is some data: > > > > We are normal here, about 1 ParNew per second. > > > > 67 2011-05-14T15:35 > > 75 2011-05-14T15:36 > > 70 2011-05-14T15:37 > > 74 2011-05-14T15:38 > > 73 2011-05-14T15:39 > > 73 2011-05-14T15:40 > > 69 2011-05-14T15:41 > > 68 2011-05-14T15:42 > > 76 2011-05-14T15:43 > > 70 2011-05-14T15:44 > > 68 2011-05-14T15:45 > > 322 2011-05-14T15:46 > > 562 2011-05-14T15:47 <------------ This is where its about 10 per > second. > > 567 2011-05-14T15:48 > > 563 2011-05-14T15:49 > > 562 2011-05-14T15:50 > > 561 2011-05-14T15:51 > > 513 2011-05-14T15:52 > > > > here is the jstack from the time of this: > > > > http://pastebin.com/RYdwH8Gp Unusually high BLOCKED threads. > > > > This is config for hbase-env: > > > > export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails > > -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError > > -Xloggc:$HBASE_HOME/logs/gc-hbase.log" > > > > Does any have any suggestions on how to manage this? > > > > -Jack > > >
