It may be sufficient for you to simply use the latest ASF release of HBase 0.90 instead of the CDH version. (Or use a snapshot from nightly.cloudera.com ?) The leaking of ZooKeeper connections suggests a particular problem. I went through the REST code on the head of the 0.90 branch and did not find it; the code is (re)using a shared Configuration instance.
If you have written client HBase java code, you should also audit your code and insure you are NOT creating a new Configuration object instance whenever you create a new HTable object, instead share a single instance. Also, insure you are not creating HTable objects without passing in a shared Configuration. Another option is to patch the HBase client, to address the likely underlying problem.See HBASE-3777 / HBASE-4508. https://issues.apache.org/jira/browse/HBASE-3777 https://issues.apache.org/jira/browse/HBASE-4508 Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) ----- Original Message ----- > From: Ronen Itkin <[email protected]> > To: [email protected] > Cc: > Sent: Sunday, October 16, 2011 8:32 PM > Subject: HBase Rest & Zookeeper issue > > Hey all! > > At first it my seem like a Zookeeper issue, but the thing is that it is very > HBase related so I figured to send it to this mailing list as well. > I am currently using Zookeeper 3.3.3-cdh3u1, installed as a distributed > cluster with 3 nodes. > Though I have installed Zookeeper separately, its only purpose right now is > to serve HBase clients (hbase-0.90.3-cdh3u1). > Seems like everything is working great for a day or two and then out the > blue, Zookeeper refuses to serve clients requests, > alerting for the wide known issue of exceeding the defined maximum > connections allowed for a client. > I changed this value to 500 and it seems that it just held on for more time > and finally reached the threshold of 500 and then stopped working. > So, I set it to 1000 and right now I am waiting to see what will happen. > > Meanwhile, by auditing the Zookeeper connections manually it seems like they > keep growing in a slow - promising pace towards the threshold of 1000 > connections. > I have tried to analyze the connections and it looks like almost all of the > connections were established from the HBase Rest service towards zookeeper. > > Can someone think of a reason why it keep growing? Does someone has any > other suggestions regarding this issue? > Are there any known issues regarding this situation, combining Rest and > Zookeeper? > > If it helps, this is my environment design: > > - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including Rest)*, > *Zookeeper01* > - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master (Including > Rest)*, *Zookeeper02* > - Server 3 - *Zookeeper03* > - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.* > > The type of operations I am committing on HBase using Zookeeper are: > > - Simple read/writes/updates using SCAN / MapReduce / Cascading .. > - Queries of the *Rest* service. > > > Thanks a lot !!! > * *Ronen Itkin >
