HI, GC log is not opened. only zk's log, pls see below: 2012-10-12 00:14:30,470 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /10.20.16.22:56954 2012-10-12 00:14:30,470 - ERROR [CommitProcessor:1:NIOServerCnxn@445] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509) at org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1545) at org.apache.zookeeper.server.DataTree.setWatches(DataTree.java:1156) at org.apache.zookeeper.server.ZKDatabase.setWatches(ZKDatabase.java:383) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:267) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73) 2012-10-12 00:14:30,470 - INFO [QuorumPeer:/0.0.0.0:2181:NIOServerCnxn@1573] - Invalid session 0x339c539ba64130b for client /10.20.16.29:41178, probably expired 2012-10-12 00:14:30,470 - ERROR [CommitProcessor:1:NIOServerCnxn@445] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509) at org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1545) at org.apache.zookeeper.server.DataTree.setWatches(DataTree.java:1156) at org.apache.zookeeper.server.ZKDatabase.setWatches(ZKDatabase.java:383) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:267) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73) 2012-10-12 00:14:30,470 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@770] - Client attempting to renew session 0x239c539ba632df9 at /10.20.16.23:37706 2012-10-12 00:14:30,470 - ERROR [CommitProcessor:1:NIOServerCnxn@445] - Unexpected Exception: java.nio.channels.CancelledKeyException
Best R. beatls On Mon, Oct 15, 2012 at 4:00 PM, Ramkrishna.S.Vasudevan < [email protected]> wrote: > Hi > > Sorry if my reply mislead you. I meant to see the GC logs that should > give > you an idea of if Full GC happened. > > Regards > Ram > > > -----Original Message----- > > From: Xiang Hua [mailto:[email protected]] > > Sent: Monday, October 15, 2012 12:42 PM > > To: [email protected] > > Subject: Re: hmaster and regionserver died > > > > We will check the zk log. > > > > On Monday, October 15, 2012, Ramkrishna.S.Vasudevan wrote: > > > > > Check your GC configurations. Seems to that a Full GC has happened > > and the > > > Zookeeper thought that to be session expiry. > > > > > > Regards > > > Ram > > > > > > > -----Original Message----- > > > > From: Xiang Hua [mailto:[email protected]] > > > > Sent: Saturday, October 13, 2012 6:20 PM > > > > To: [email protected] > > > > Subject: hmaster and regionserver died > > > > > > > > Hi, > > > > the HMaster died as well as regionservers, below is hmaster's > > log. > > > > could > > > > you please find what's problem? > > > > > > > > > > > > 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > > > connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/ > > > > 10.20.16.34:2181, initiating session > > > > 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn: > > Session > > > > establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase- > > stor- > > > > 3/ > > > > 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout > > = > > > > 40000 > > > > 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn: > > Client > > > > session timed out, have not heard from server in 15046ms for > > sessionid > > > > 0x239c539ba630001, closing socket connection and attempting > > reconnect > > > > 2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn: > > Opening > > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > > > 10.20.16.33:2181 > > > > 2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn: > > Client > > > > session timed out, have not heard from server in 15245ms for > > sessionid > > > > 0x139c539bc090003, closing socket connection and attempting > > reconnect > > > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: > > Opening > > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > > > 10.20.16.33:2181 > > > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > > > 10.20.16.33:2181, initiating session > > > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: > > > > EventThread > > > > shut down > > > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: > > Unable to > > > > reconnect to ZooKeeper service, session 0x139c539bc090003 has > > expired, > > > > closing socket connection > > > > 2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > > > 10.20.16.33:2181, initiating session > > > > 2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn: > > Session > > > > 0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor- > > 2/ > > > > 10.20.16.33:2181, unexpected error, closing socket connection and > > > > attempting reconnect > > > > java.io.IOException: Connection reset by peer > > > > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > > > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > > > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218) > > > > at sun.nio.ch.IOUtil.read(IOUtil.java:186) > > > > at > > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359) > > > > at > > > > > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859) > > > > at > > > > > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157) > > > > 2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn: > > Opening > > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/ > > > > 10.20.16.34:2181 > > > > 2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn: > > Client > > > > session timed out, have not heard from server in 14007ms for > > sessionid > > > > 0x239c539ba630001, closing socket connection and attempting > > reconnect > > > > 2012-10-12 00:14:41,592 INFO org.apache.zookeeper.ClientCnxn: > > Opening > > > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/ > > > > 10.20.16.32:2181 > > > > 2012-10-12 00:14:46,186 INFO org.apache.zookeeper.ClientCnxn: > > Client > > > > session timed out, have not heard from server in 26666ms for > > sessionid > > > > >
