What is the size of your new gen? Is it growing? Does it level off? St.Ack
On Mon, May 16, 2011 at 12:39 PM, Jack Levin <[email protected]> wrote: > There HEAP would be 8G used out of 12G total. The gc-log would be > full of ParNew, no FULL GC at all, ParNew just starts at hight rate > (10-15 lines per second). Even if you remove queries, the JVM will > not recover, and user CPU will remain 100%. Only thing left to do is > to kill the process. We are not writing in nearly as fast to case > such an issue. This looks like a systemic problem, e.g. low enough > load when it comes to QPS, and the problem effectively kills HBASE. > There should be a way to tune things so that ParNew does not jump to > an inappropriate levels. > > -Jack > > On Mon, May 16, 2011 at 12:02 PM, Stack <[email protected]> wrote: >> On Sun, May 15, 2011 at 5:37 PM, Jack Levin <[email protected]> wrote: >>> I've added occupancy: export HBASE_OPTS="$HBASE_OPTS -verbose:gc >>> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails >>> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError >>> -Xloggc:$HBASE_HOME/logs/gc-hbase.log" >>> >> >> Does the big CPU usage correlate to a Full GC? Do you notice that in >> your logs Jack? >> >> I took a look at your thread dumps. Nothing untoward (blocking on hdfs >> access). >> >> St.Ack >> >
