We are running Quorum of three zookeepers to connect to our hadoop 2.6.0 setup. However surprisingly if one of the zookeeper goes down our system goes. Below is a log from our wildfly server, which stopped responding as one of the zookeeper down and it only tried to connect to that particular zookeeper. How do we ensure that if one goes down then also our system should redirect to next one and keep going?
16:38:12,838 ERROR [org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper] (default task-27) ZooKeeper getData failed after 4 attempts 16:38:12,838 ERROR [org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher] (default task-27) hconnection-0x2f95012, quorum=zookeeper1:2181 <http://us-east-1-zookeeper-aws-1.icare.com:2181/>, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) [zookeeper-3.4.5.jar:3.4.5-1392090] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [zookeeper-3.4.5.jar:3.4.5-1392090] at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151) [zookeeper-3.4.5.jar:3.4.5-1392090] at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:337) [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:683) [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] at org.apache.hadoop.hbase.zookeeper.ZKUtil.blockUntilAvailable(ZKUtil.java:1835) [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2] at org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:183) [hbase-client-0.98.1-hadoop2.jar:0.98.1-hadoop2]
