GC and High CPU

[Jack Levin]

Greetings all -- We had some issues with huge GC counts that drive CPU
to 100% and rendering RS basically dead, here is some data:

 We are normal here, about 1 ParNew per second.

    67 2011-05-14T15:35
     75 2011-05-14T15:36
     70 2011-05-14T15:37
     74 2011-05-14T15:38
     73 2011-05-14T15:39
     73 2011-05-14T15:40
     69 2011-05-14T15:41
     68 2011-05-14T15:42
     76 2011-05-14T15:43
     70 2011-05-14T15:44
     68 2011-05-14T15:45
    322 2011-05-14T15:46
    562 2011-05-14T15:47  <------------ This is where its about 10 per second.
    567 2011-05-14T15:48
    563 2011-05-14T15:49
    562 2011-05-14T15:50
    561 2011-05-14T15:51
    513 2011-05-14T15:52

here is the jstack from the time of this:

http://pastebin.com/RYdwH8Gp  Unusually high BLOCKED threads.

This is config for hbase-env:

export HBASE_OPTS="$HBASE_OPTS -verbose:gc  -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
-Xloggc:$HBASE_HOME/logs/gc-hbase.log"

Does any have any suggestions on how to manage this?

-Jack