There HEAP would be 8G used out of 12G total. The gc-log would be full of ParNew, no FULL GC at all, ParNew just starts at hight rate (10-15 lines per second). Even if you remove queries, the JVM will not recover, and user CPU will remain 100%. Only thing left to do is to kill the process. We are not writing in nearly as fast to case such an issue. This looks like a systemic problem, e.g. low enough load when it comes to QPS, and the problem effectively kills HBASE. There should be a way to tune things so that ParNew does not jump to an inappropriate levels.
-Jack On Mon, May 16, 2011 at 12:02 PM, Stack <[email protected]> wrote: > On Sun, May 15, 2011 at 5:37 PM, Jack Levin <[email protected]> wrote: >> I've added occupancy: export HBASE_OPTS="$HBASE_OPTS -verbose:gc >> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails >> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError >> -Xloggc:$HBASE_HOME/logs/gc-hbase.log" >> > > Does the big CPU usage correlate to a Full GC? Do you notice that in > your logs Jack? > > I took a look at your thread dumps. Nothing untoward (blocking on hdfs > access). > > St.Ack >
