Pardon for any confusion (too late? :-) ) but this looks like a known HBase client bug.
HBASE-3777 (https://issues.apache.org/jira/browse/HBASE-3777) has background. Here is what I recommend: 1) Use a build from the head of 0.90 branch, not a CDH build (I have no idea what revision that corresponds to)* and if that is not sufficient 2) We will need to apply HBASE-3777 to 0.90 branch. * - I reviewed the REST code on the 0.90 branch. It takes care to reuse a Configuration object and a common HTable pool for all table access. - Andy ----- Original Message ----- > From: Ronen Itkin <[email protected]> > To: [email protected]; Andrew Purtell <[email protected]> > Cc: > Sent: Monday, October 17, 2011 6:12 PM > Subject: Re: HBase Rest & Zookeeper issue > > Hey, > > Another update, I had just stopped the Rest Service and the number of > zookeeper client connections from that server dropped from 513 to 3. > > > > On Mon, Oct 17, 2011 at 11:57 AM, Ronen Itkin <[email protected]> wrote: > >> Hey Andrew, >> >> Thanks for the response, >> Please note that the majority of zookeeper client connections are being >> made from the Rest service itself and not from any other Java HBase Client, >> Moreover, Rest Client address the Rest Service itself and not zookeepeer >> (while The rest Service works with zookeeper-->hbase at the background). >> So, I guess its not something in our code, but something regarding Rest >> and/or ZooKeeper operations. >> >> To be certain that the connections grow mainly because the Rest Service and >> not because of other HBase Java Clients, I will shut it down for 5-6 Hours. >> >> Can you please aleborate more about using a "shared single > instance" ? is >> that something configurable for the Rest implementation? >> Is there any possibility that a Java HBase client is not using cache for >> zookeeper queries? >> >> Thanks, >> >> >> On Mon, Oct 17, 2011 at 4:39 AM, Andrew Purtell > <[email protected]>wrote: >> >>> It may be sufficient for you to simply use the latest ASF release of > HBase >>> 0.90 instead of the CDH version. (Or use a snapshot from >>> nightly.cloudera.com ?) The leaking of ZooKeeper connections suggests a >>> particular problem. I went through the REST code on the head of the > 0.90 >>> branch and did not find it; the code is (re)using a shared >>> Configuration instance. >>> >>> If you have written client HBase java code, you should also audit your >>> code and insure you are NOT creating a new Configuration object > instance >>> whenever you create a new HTable object, instead share a single > instance. >>> Also, insure you are not creating HTable objects without passing in a > shared >>> Configuration. >>> >>> >>> Another option is to patch the HBase client, to address the likely >>> underlying problem.See HBASE-3777 / HBASE-4508. >>> >>> https://issues.apache.org/jira/browse/HBASE-3777 >>> https://issues.apache.org/jira/browse/HBASE-4508 >>> >>> Best regards, >>> >>> >>> - Andy >>> >>> Problems worthy of attack prove their worth by hitting back. - Piet > Hein >>> (via Tom White) >>> >>> >>> ----- Original Message ----- >>> > From: Ronen Itkin <[email protected]> >>> > To: [email protected] >>> > Cc: >>> > Sent: Sunday, October 16, 2011 8:32 PM >>> > Subject: HBase Rest & Zookeeper issue >>> > >>> > Hey all! >>> > >>> > At first it my seem like a Zookeeper issue, but the thing is that > it is >>> very >>> > HBase related so I figured to send it to this mailing list as > well. >>> > I am currently using Zookeeper 3.3.3-cdh3u1, installed as a > distributed >>> > cluster with 3 nodes. >>> > Though I have installed Zookeeper separately, its only purpose > right now >>> is >>> > to serve HBase clients (hbase-0.90.3-cdh3u1). >>> > Seems like everything is working great for a day or two and then > out the >>> > blue, Zookeeper refuses to serve clients requests, >>> > alerting for the wide known issue of exceeding the defined maximum >>> > connections allowed for a client. >>> > I changed this value to 500 and it seems that it just held on for > more >>> time >>> > and finally reached the threshold of 500 and then stopped working. >>> > So, I set it to 1000 and right now I am waiting to see what will > happen. >>> > >>> > Meanwhile, by auditing the Zookeeper connections manually it seems > like >>> they >>> > keep growing in a slow - promising pace towards the threshold of > 1000 >>> > connections. >>> > I have tried to analyze the connections and it looks like almost > all of >>> the >>> > connections were established from the HBase Rest service towards >>> zookeeper. >>> > >>> > Can someone think of a reason why it keep growing? Does someone > has any >>> > other suggestions regarding this issue? >>> > Are there any known issues regarding this situation, combining > Rest and >>> > Zookeeper? >>> > >>> > If it helps, this is my environment design: >>> > >>> > - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master > (Including >>> Rest)*, >>> > *Zookeeper01* >>> > - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master > (Including >>> > Rest)*, *Zookeeper02* >>> > - Server 3 - *Zookeeper03* >>> > - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region > Server.* >>> > >>> > The type of operations I am committing on HBase using Zookeeper > are: >>> > >>> > - Simple read/writes/updates using SCAN / MapReduce / Cascading > .. >>> > - Queries of the *Rest* service. >>> > >>> > >>> > Thanks a lot !!! >>> > * *Ronen Itkin >>> > >>> >> >> >> >> -- >> * >> Ronen Itkin* >> Taykey | www.taykey.com >> >> > > > -- > * > Ronen Itkin* > Taykey | www.taykey.com >
