So. I have restarted everything and I found the issue... I ran the code above from Eclipse where I have HBase trunk. Cluster is running with HBase 0.94.2.
Conclusion: HBase Trunk is not compatible with 0.94.2? Anyway, now it's working again... JM 2012/11/15, Jean-Marc Spaggiari <[email protected]>: > HBase stop/start did not helped... > ZooKeeper stop/start did not helped.. > > I reduced the code to the minimum: > config = HBaseConfiguration.create(); > config.set("hbase.zookeeper.quorum", "cube"); > new HBaseAdmin(config); > > But still getting the exception. > > On the same machine, bin/hbase zkcli is working fine giving me a > result to get /hbase. > bin/hbase shell is working fine too. > > I'm a bit lost here. I tryied to google the issue with no much > results. I will continue to dig... > > JM > > 2012/11/15, Jean-Marc Spaggiari <[email protected]>: >> Hi, >> >> I ran a 30h MapReduce job, and now I'm not able to connect anymore to >> my HBase cluster. >> >> The MapReduce was configured on ReadOnly mode. So only the log table >> received data. Everything else was just ready. >> >> Today I killed the job to replace one of the servers which is too slow >> and now I'm not able to connect to ZooKeeper anymore. >> >> Below is the stack trace, and at the bottom is the ZKDump. >> >> I think if I restart everything it should be working, but I'm >> wondering if there is any information on this situation which might >> help to prevent this to happend in the futur? I might be able to >> reproduce that since I can re-run the job almost anytime. >> >> I don't seems to have to many connections. The HBase shell is replying >> correctly. It's really only the Java application using ZooKeeper which >> is not working. >> >> I'm running HBase 0.94.2, ZK 3.4.3, Hadoop 1.0.3 all installed separatly. >> >> JM >> >> 2012-11-15 18:09:33,684 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:33,810 [main] WARN >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly >> transient ZooKeeper exception: >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid >> 2012-11-15 18:09:34,800 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:35,902 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:36,003 [main] WARN >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly >> transient ZooKeeper exception: >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid >> 2012-11-15 18:09:37,004 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:38,106 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:39,209 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:40,311 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:40,412 [main] WARN >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly >> transient ZooKeeper exception: >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid >> 2012-11-15 18:09:41,414 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:42,516 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:43,618 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:44,720 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:45,822 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:46,924 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:48,027 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:49,129 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:49,230 [main] WARN >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly >> transient ZooKeeper exception: >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid >> 2012-11-15 18:09:49,231 [main] ERROR >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - ZooKeeper >> exists failed after 3 retries >> 2012-11-15 18:09:49,231 [main] WARN >> org.apache.hadoop.hbase.zookeeper.ZKUtil - hconnection 0x1781735b >> Unable to set watcher on znode (/hbase/hbaseid) >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) >> at >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173) >> at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:262) >> at >> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:63) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:645) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:620) >> at >> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:215) >> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:143) >> 2012-11-15 18:09:49,232 [main] ERROR >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher - hconnection >> 0x1781735b Received unexpected KeeperException, re-throwing exception >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) >> at >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173) >> at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:262) >> at >> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:63) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:645) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:620) >> at >> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:215) >> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:143) >> 2012-11-15 18:09:49,233 [main] WARN >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation >> - Can't retrieve clusterId from Zookeeper >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/hbaseid >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) >> at >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173) >> at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:262) >> at >> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:63) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:645) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:620) >> at >> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:215) >> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:143) >> Creating table_page_crc. 2012-11-15 18:09:49,255 >> [main-SendThread(cube:21818)] WARN org.apache.zookeeper.ClientCnxn - >> Session 0x0 for server null, unexpected error, closing socket >> connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:49,357 [main] WARN >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly >> transient ZooKeeper exception: >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >> 2012-11-15 18:09:50,231 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:50,358 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:51,334 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:51,460 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:51,561 [main] WARN >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly >> transient ZooKeeper exception: >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >> 2012-11-15 18:09:52,436 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:52,562 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:53,538 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:53,664 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:54,640 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:54,766 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:55,742 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:55,868 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:55,969 [main] WARN >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly >> transient ZooKeeper exception: >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >> 2012-11-15 18:09:56,844 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:56,970 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:57,946 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:58,072 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:59,048 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:09:59,174 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:00,150 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:00,276 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:01,253 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:01,378 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:02,355 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:02,480 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:03,457 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:03,582 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:04,559 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:04,684 [main-SendThread(cube:21818)] WARN >> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> unexpected error, closing socket connection and attempting reconnect >> java.net.ConnectException: Connexion refuse >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> 2012-11-15 18:10:04,785 [main] WARN >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly >> transient ZooKeeper exception: >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >> 2012-11-15 18:10:04,785 [main] ERROR >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - ZooKeeper >> exists failed after 3 retries >> 2012-11-15 18:10:04,785 [main] WARN >> org.apache.hadoop.hbase.zookeeper.ZKUtil - >> catalogtracker-on-hconnection 0x1781735b Unable to set watcher on >> znode /hbase/root-region-server >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) >> at >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173) >> at >> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:85) >> at >> org.apache.hadoop.hbase.catalog.CatalogTracker.start(CatalogTracker.java:242) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:177) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:225) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:241) >> 2012-11-15 18:10:04,786 [main] ERROR >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher - >> catalogtracker-on-hconnection 0x1781735b Received unexpected >> KeeperException, re-throwing exception >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) >> at >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173) >> at >> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:85) >> at >> org.apache.hadoop.hbase.catalog.CatalogTracker.start(CatalogTracker.java:242) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:177) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:225) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:241) >> 2012-11-15 18:10:04,787 [main] FATAL >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation >> - Unexpected exception during initialization, aborting >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) >> at >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173) >> at >> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:85) >> at >> org.apache.hadoop.hbase.catalog.CatalogTracker.start(CatalogTracker.java:242) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:177) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:225) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:241) >> java.io.IOException: Attempt to start root/meta tracker failed. >> at >> org.apache.hadoop.hbase.catalog.CatalogTracker.start(CatalogTracker.java:247) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:177) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:225) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:241) >> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021) >> at >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:173) >> at >> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:85) >> at >> org.apache.hadoop.hbase.catalog.CatalogTracker.start(CatalogTracker.java:242) >> ... 5 more >> >> >> >> >> >> HBase is rooted at /hbase >> Active master address: node3,60000,1352914424646 >> Backup master addresses: >> Region server holding ROOT: node5,60020,1352914415945 >> Region servers: >> latitude,60020,1352914441019 >> node6,60020,1352914439333 >> node5,60020,1352914415945 >> node2,60020,1353003249113 >> phenom,60020,1352914434321 >> node4,60020,1353009796200 >> node3,60020,1352914437336 >> node1,60020,1352914438812 >> Quorum Server Statistics: >> cube:2181 >> Zookeeper version: 3.4.3-1240972, built on 02/06/2012 10:48 GMT >> Clients: >> /192.168.23.7:32878[1](queued=0,recved=36,sent=36) >> /192.168.23.5:49773[1](queued=0,recved=25,sent=25) >> /192.168.23.7:32875[1](queued=0,recved=25,sent=25) >> /192.168.23.5:49774[1](queued=0,recved=25,sent=25) >> /192.168.23.8:49753[1](queued=0,recved=25,sent=25) >> /192.168.23.10:55277[1](queued=0,recved=25,sent=25) >> /192.168.23.7:32877[1](queued=0,recved=25,sent=25) >> /192.168.23.6:44692[1](queued=0,recved=25,sent=25) >> /192.168.23.4:52766[1](queued=0,recved=25,sent=25) >> /192.168.23.7:32876[1](queued=0,recved=25,sent=25) >> /192.168.23.2:35018[1](queued=0,recved=25,sent=25) >> /192.168.23.8:49752[1](queued=0,recved=25,sent=25) >> /192.168.23.7:32899[0](queued=0,recved=1,sent=0) >> /192.168.23.7:32874[1](queued=0,recved=25,sent=25) >> /192.168.23.10:55275[1](queued=0,recved=25,sent=25) >> /192.168.23.9:37189[1](queued=0,recved=25,sent=25) >> /192.168.23.2:35019[1](queued=0,recved=25,sent=25) >> /192.168.23.9:37190[1](queued=0,recved=25,sent=25) >> /192.168.23.4:52765[1](queued=0,recved=25,sent=25) >> /192.168.23.6:44693[1](queued=0,recved=25,sent=25) >> >> Latency min/avg/max: 0/0/18 >> Received: 487 >> Sent: 486 >> Outstanding: 0 >> Zxid: 0xc0002bbdf >> Mode: standalone >> Node count: 31 >> >
