Session expired usually results from a long GC that exceeds the zookeeper.session.timeout.
> 2012-07-01 18:20:00,961 FATAL > org.apache.hadoop.hbase.master.HMaster:master:60000-0x238444cf77e0000 > master:60000-0x238444cf77e0000 received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode = Session expired See http://hbase.apache.org/book.html#trouble.log.gc --Suraj On Sun, Jul 1, 2012 at 6:34 PM, Jay Wilson <[email protected]> wrote: > I assume a timer may need to be increased based on these messages. Any > thoughts as to which one or is it something else? > > 2012-07-01 18:17:16,461 INFO > org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. > servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=0 > 2012-07-01 18:17:16,466 DEBUG > org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) > and gc'd 0 unreferenced parent region(s) > 2012-07-01 18:20:00,281 INFO org.apache.zookeeper.ClientCnxn: Unable to > read additional data from server sessionid 0x38444cf77f0000, likely > server has closed socket, closing socket connection and attempting reconnect > 2012-07-01 18:20:00,281 INFO org.apache.zookeeper.ClientCnxn: Unable to > read additional data from server sessionid 0x238444cf77e0000, likely > server has closed socket, closing socket connection and attempting reconnect > 2012-07-01 18:20:00,384 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: > hconnection-0x38444cf77f0000 Unable to set watcher on znode /hbase/master > org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:178) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:294) > at > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) > 2012-07-01 18:20:00,386 ERROR > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > hconnection-0x38444cf77f0000 Received unexpected KeeperException, > re-throwing exception > org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:178) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:294) > at > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) > 2012-07-01 18:20:00,387 INFO > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > This client just lost it's session with ZooKeeper, trying to reconnect. > 2012-07-01 18:20:00,387 INFO > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Trying to reconnect to zookeeper. > 2012-07-01 18:20:00,957 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server devrackA-05/172.18.0.6:2181 > 2012-07-01 18:20:00,958 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to devrackA-05/172.18.0.6:2181, initiating session > 2012-07-01 18:20:00,961 INFO org.apache.zookeeper.ClientCnxn: Unable to > reconnect to ZooKeeper service, session 0x238444cf77e0000 has expired, > closing socket connection > 2012-07-01 18:20:00,961 FATAL org.apache.hadoop.hbase.master.HMaster: > master:60000-0x238444cf77e0000 master:60000-0x238444cf77e0000 received > expired from ZooKeeper, aborting > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279) > at > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) > 2012-07-01 18:20:00,961 INFO org.apache.hadoop.hbase.master.HMaster: > Aborting > 2012-07-01 18:20:00,961 INFO org.apache.zookeeper.ClientCnxn: > EventThread shut down > > --- > Jay Wilson
