Sorry for the delay. Here is the link to the GC log.
http://pastebin.com/m13b52962 At line 1867, you will see a big number 2170 My GC configuration is export HBASE_OPTS="-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:ParallelGCThreads=8" My GC logging configuration is: -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/data/hbase_logs/gc-hbase.log Have a wonderful weekend! Best, zhenyu On Sat, Nov 14, 2009 at 5:56 PM, stack <[email protected]> wrote: > So, tell us more about your GC configuration and please post your GC log > somewhere. A few of the lads here on this list are good at looking at GC > logs. > Thanks, > St.Ack > > On Sat, Nov 14, 2009 at 1:20 PM, Zhenyu Zhong <[email protected] > >wrote: > > > Stack, > > > > I am very appreciated for your help. > > I went back to check the GC log. > > I found one entry that is > > 8677.032: [CMS-concurrent-mark: 1.547/2170.359 secs] [Times: user=50.29 > > sys=3.65, real=2170.49 secs] > > I couldn't tell what time this entry was logged. > > > > This is the only entry I found that has such huge number of GC time. > > I also saw a few long GC time around 200 secs. (should be within the > > session > > timeout) > > > > Next, I need to find a way to correlate the timestamp with a friendly > > dateformat on each GC log entry. > > > > Hopefully this is the reason. > > > > Best, > > zhenyu > > > > > > > > > > On Sat, Nov 14, 2009 at 3:03 PM, stack <[email protected]> wrote: > > > > > On Sat, Nov 14, 2009 at 8:31 AM, Zhenyu Zhong <[email protected] > > > >wrote: > > > > > > .... > > > > > > > 2009-11-14 16:00:12,775 DEBUG org.apache.zookeeper.ClientCnxn: Got > ping > > > > response for sessionid:0x424f0ed01ea00a6 after 0ms > > > > 2009-11-14 16:00:28,141 DEBUG org.apache.zookeeper.ClientCnxn: Got > ping > > > > response for sessionid:0x424f0ed01ea00a6 after 2010ms > > > > 2009-11-14 16:01:10,378 WARN org.apache.zookeeper.ClientCnxn: > Exception > > > > closing session 0x424f0ed01ea00a6 to > > sun.nio.ch.selectionkeyi...@39fbb2d6 > > > > java.io.IOException: Read error rc = -1 > java.nio.DirectByteBuffer[pos=0 > > > > lim=4 cap=4] > > > > at > > > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701) > > > > > > > > > > Anything happening on the machine at this time? The ping time is > > starting > > > to climb... was 0ms for a long time and the last log is > 2seconds. > > Whats > > > in the GC logs? Anything else running on this machine? > > > St.Ack > > > > > >
