Hey all! How are you? I am currently using Zookeeper 3.3.3-cdh3u1, installed as a distributed cluster with 3 nodes. Though I have installed Zookeeper separately, its only purpose right now is to serve HBase clients (hbase-0.90.3-cdh3u1). Seems like everything is working great for a day or two and that out the blue, Zookeeper refuses to serve clients requests, alerting for the wide known issue of exceeding the defined maximum connections allowed for a client. I changed this value to 500 and it seems that it just held on for more time and finally reached the threshold of 500 and then stopped working. So, I set it to 1000 and right now I am waiting to see what will happen.
Meanwhile, by auditing the Zookeeper connections manually it seems like they keep growing in a slow - promising pace towards the threshold of 1000 connections. Can someone think of a reason why it happens? Why it keep growing? Does someone has any other suggestions regarding this issue? What is the recommended value for Zookeeper maximum connections per client? can it overload the server if set to a high value? what is a high value in this case? If it helps, this is my environment design: - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including Rest)*, *Zookeeper01* - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master (Including Rest)*, *Zookeeper02* - Server 3 - *Zookeeper03* - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.* The type of operations I am committing on HBase using Zookeeper are: - Simple read/writes/updates using SCAN / MapReduce / Cascading .. - Queries of the Rest service. Thanks a lot !!! * *Ronen Itkin <http://www.taykey.com/>
