Seems like a hbase problem to me... Zk doesn't randomly create or hang on to client connections. Did you ask on the hbase lists?
C On Sunday, October 16, 2011, Ronen Itkin <[email protected]> wrote: > Hey all! > How are you? > > I am currently using Zookeeper 3.3.3-cdh3u1, installed as a distributed > cluster with 3 nodes. > Though I have installed Zookeeper separately, its only purpose right now is > to serve HBase clients (hbase-0.90.3-cdh3u1). > Seems like everything is working great for a day or two and that out the > blue, Zookeeper refuses to serve clients requests, > alerting for the wide known issue of exceeding the defined maximum > connections allowed for a client. > I changed this value to 500 and it seems that it just held on for more time > and finally reached the threshold of 500 and then stopped working. > So, I set it to 1000 and right now I am waiting to see what will happen. > > Meanwhile, by auditing the Zookeeper connections manually it seems like they > keep growing in a slow - promising pace towards the threshold of 1000 > connections. > > Can someone think of a reason why it happens? Why it keep growing? Does > someone has any other suggestions regarding this issue? > What is the recommended value for Zookeeper maximum connections per client? > can it overload the server if set to a high value? what is a high value in > this case? > > If it helps, this is my environment design: > > - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including Rest)*, > *Zookeeper01* > - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master (Including > Rest)*, *Zookeeper02* > - Server 3 - *Zookeeper03* > - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.* > > The type of operations I am committing on HBase using Zookeeper are: > > - Simple read/writes/updates using SCAN / MapReduce / Cascading .. > - Queries of the Rest service. > > > Thanks a lot !!! > * *Ronen Itkin > > <http://www.taykey.com/> >
