Hey all! At first it my seem like a Zookeeper issue, but the thing is that it is very HBase related so I figured to send it to this mailing list as well. I am currently using Zookeeper 3.3.3-cdh3u1, installed as a distributed cluster with 3 nodes. Though I have installed Zookeeper separately, its only purpose right now is to serve HBase clients (hbase-0.90.3-cdh3u1). Seems like everything is working great for a day or two and then out the blue, Zookeeper refuses to serve clients requests, alerting for the wide known issue of exceeding the defined maximum connections allowed for a client. I changed this value to 500 and it seems that it just held on for more time and finally reached the threshold of 500 and then stopped working. So, I set it to 1000 and right now I am waiting to see what will happen.
Meanwhile, by auditing the Zookeeper connections manually it seems like they keep growing in a slow - promising pace towards the threshold of 1000 connections. I have tried to analyze the connections and it looks like almost all of the connections were established from the HBase Rest service towards zookeeper. Can someone think of a reason why it keep growing? Does someone has any other suggestions regarding this issue? Are there any known issues regarding this situation, combining Rest and Zookeeper? If it helps, this is my environment design: - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including Rest)*, *Zookeeper01* - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master (Including Rest)*, *Zookeeper02* - Server 3 - *Zookeeper03* - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.* The type of operations I am committing on HBase using Zookeeper are: - Simple read/writes/updates using SCAN / MapReduce / Cascading .. - Queries of the *Rest* service. Thanks a lot !!! * *Ronen Itkin
