We will check the zk log. On Monday, October 15, 2012, Ramkrishna.S.Vasudevan wrote:
> Check your GC configurations. Seems to that a Full GC has happened and the > Zookeeper thought that to be session expiry. > > Regards > Ram > > > -----Original Message----- > > From: Xiang Hua [mailto:[email protected]] > > Sent: Saturday, October 13, 2012 6:20 PM > > To: [email protected] > > Subject: hmaster and regionserver died > > > > Hi, > > the HMaster died as well as regionservers, below is hmaster's log. > > could > > you please find what's problem? > > > > > > 2012-10-12 00:14:19,444 INFO org.apache.zookeeper.ClientCnxn: Socket > > connection established to bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/ > > 10.20.16.34:2181, initiating session > > 2012-10-12 00:14:19,520 INFO org.apache.zookeeper.ClientCnxn: Session > > establishment complete on server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor- > > 3/ > > 10.20.16.34:2181, sessionid = 0x139c539bc090002, negotiated timeout = > > 40000 > > 2012-10-12 00:14:23,738 INFO org.apache.zookeeper.ClientCnxn: Client > > session timed out, have not heard from server in 15046ms for sessionid > > 0x239c539ba630001, closing socket connection and attempting reconnect > > 2012-10-12 00:14:24,246 INFO org.apache.zookeeper.ClientCnxn: Opening > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > 10.20.16.33:2181 > > 2012-10-12 00:14:25,173 INFO org.apache.zookeeper.ClientCnxn: Client > > session timed out, have not heard from server in 15245ms for sessionid > > 0x139c539bc090003, closing socket connection and attempting reconnect > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Opening > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > 10.20.16.33:2181 > > 2012-10-12 00:14:25,328 INFO org.apache.zookeeper.ClientCnxn: Socket > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > 10.20.16.33:2181, initiating session > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: > > EventThread > > shut down > > 2012-10-12 00:14:25,507 INFO org.apache.zookeeper.ClientCnxn: Unable to > > reconnect to ZooKeeper service, session 0x139c539bc090003 has expired, > > closing socket connection > > 2012-10-12 00:14:27,247 INFO org.apache.zookeeper.ClientCnxn: Socket > > connection established to bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > 10.20.16.33:2181, initiating session > > 2012-10-12 00:14:27,248 WARN org.apache.zookeeper.ClientCnxn: Session > > 0x239c539ba630001 for server bj-ecsxhm4f3I-r3-5-r810-3-hbase-stor-2/ > > 10.20.16.33:2181, unexpected error, closing socket connection and > > attempting reconnect > > java.io.IOException: Connection reset by peer > > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218) > > at sun.nio.ch.IOUtil.read(IOUtil.java:186) > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359) > > at > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:859) > > at > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157) > > 2012-10-12 00:14:28,026 INFO org.apache.zookeeper.ClientCnxn: Opening > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-2-hbase-stor-3/ > > 10.20.16.34:2181 > > 2012-10-12 00:14:41,359 INFO org.apache.zookeeper.ClientCnxn: Client > > session timed out, have not heard from server in 14007ms for sessionid > > 0x239c539ba630001, closing socket connection and attempting reconnect > > 2012-10-12 00:14:41,592 INFO org.apache.zookeeper.ClientCnxn: Opening > > socket connection to server bj-ecsxhm4f3I-r3-5-r810-4-hbase-stor-1/ > > 10.20.16.32:2181 > > 2012-10-12 00:14:46,186 INFO org.apache.zookeeper.ClientCnxn: Client > > session timed out, have not heard from server in 26666ms for sessionid >
