For item 2 below, patch for HBASE-4508 (Backport HBASE-3777 to 0.90 branch<https://issues.apache.org/jira/browse/HBASE-4508>) has been ready for review / verification.
Cheers On Mon, Oct 17, 2011 at 5:44 AM, Andrew Purtell <[email protected]> wrote: > Pardon for any confusion (too late? :-) ) but this looks like a known HBase > client bug. > > HBASE-3777 (https://issues.apache.org/jira/browse/HBASE-3777) has > background. > > Here is what I recommend: > > 1) Use a build from the head of 0.90 branch, not a CDH build (I have no > idea what revision that corresponds to)* > > and if that is not sufficient > > 2) We will need to apply HBASE-3777 to 0.90 branch. > > * - I reviewed the REST code on the 0.90 branch. It takes care to reuse a > Configuration object and a common HTable pool for all table access. > > - Andy > > > ----- Original Message ----- > > From: Ronen Itkin <[email protected]> > > To: [email protected]; Andrew Purtell <[email protected]> > > Cc: > > Sent: Monday, October 17, 2011 6:12 PM > > Subject: Re: HBase Rest & Zookeeper issue > > > > Hey, > > > > Another update, I had just stopped the Rest Service and the number of > > zookeeper client connections from that server dropped from 513 to 3. > > > > > > > > On Mon, Oct 17, 2011 at 11:57 AM, Ronen Itkin <[email protected]> wrote: > > > >> Hey Andrew, > >> > >> Thanks for the response, > >> Please note that the majority of zookeeper client connections are being > >> made from the Rest service itself and not from any other Java HBase > Client, > >> Moreover, Rest Client address the Rest Service itself and not > zookeepeer > >> (while The rest Service works with zookeeper-->hbase at the > background). > >> So, I guess its not something in our code, but something regarding Rest > >> and/or ZooKeeper operations. > >> > >> To be certain that the connections grow mainly because the Rest Service > and > >> not because of other HBase Java Clients, I will shut it down for 5-6 > Hours. > >> > >> Can you please aleborate more about using a "shared single > > instance" ? is > >> that something configurable for the Rest implementation? > >> Is there any possibility that a Java HBase client is not using cache > for > >> zookeeper queries? > >> > >> Thanks, > >> > >> > >> On Mon, Oct 17, 2011 at 4:39 AM, Andrew Purtell > > <[email protected]>wrote: > >> > >>> It may be sufficient for you to simply use the latest ASF release of > > HBase > >>> 0.90 instead of the CDH version. (Or use a snapshot from > >>> nightly.cloudera.com ?) The leaking of ZooKeeper connections suggests > a > >>> particular problem. I went through the REST code on the head of the > > 0.90 > >>> branch and did not find it; the code is (re)using a shared > >>> Configuration instance. > >>> > >>> If you have written client HBase java code, you should also audit your > >>> code and insure you are NOT creating a new Configuration object > > instance > >>> whenever you create a new HTable object, instead share a single > > instance. > >>> Also, insure you are not creating HTable objects without passing in a > > shared > >>> Configuration. > >>> > >>> > >>> Another option is to patch the HBase client, to address the likely > >>> underlying problem.See HBASE-3777 / HBASE-4508. > >>> > >>> https://issues.apache.org/jira/browse/HBASE-3777 > >>> https://issues.apache.org/jira/browse/HBASE-4508 > >>> > >>> Best regards, > >>> > >>> > >>> - Andy > >>> > >>> Problems worthy of attack prove their worth by hitting back. - Piet > > Hein > >>> (via Tom White) > >>> > >>> > >>> ----- Original Message ----- > >>> > From: Ronen Itkin <[email protected]> > >>> > To: [email protected] > >>> > Cc: > >>> > Sent: Sunday, October 16, 2011 8:32 PM > >>> > Subject: HBase Rest & Zookeeper issue > >>> > > >>> > Hey all! > >>> > > >>> > At first it my seem like a Zookeeper issue, but the thing is that > > it is > >>> very > >>> > HBase related so I figured to send it to this mailing list as > > well. > >>> > I am currently using Zookeeper 3.3.3-cdh3u1, installed as a > > distributed > >>> > cluster with 3 nodes. > >>> > Though I have installed Zookeeper separately, its only purpose > > right now > >>> is > >>> > to serve HBase clients (hbase-0.90.3-cdh3u1). > >>> > Seems like everything is working great for a day or two and then > > out the > >>> > blue, Zookeeper refuses to serve clients requests, > >>> > alerting for the wide known issue of exceeding the defined maximum > >>> > connections allowed for a client. > >>> > I changed this value to 500 and it seems that it just held on for > > more > >>> time > >>> > and finally reached the threshold of 500 and then stopped working. > >>> > So, I set it to 1000 and right now I am waiting to see what will > > happen. > >>> > > >>> > Meanwhile, by auditing the Zookeeper connections manually it seems > > like > >>> they > >>> > keep growing in a slow - promising pace towards the threshold of > > 1000 > >>> > connections. > >>> > I have tried to analyze the connections and it looks like almost > > all of > >>> the > >>> > connections were established from the HBase Rest service towards > >>> zookeeper. > >>> > > >>> > Can someone think of a reason why it keep growing? Does someone > > has any > >>> > other suggestions regarding this issue? > >>> > Are there any known issues regarding this situation, combining > > Rest and > >>> > Zookeeper? > >>> > > >>> > If it helps, this is my environment design: > >>> > > >>> > - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master > > (Including > >>> Rest)*, > >>> > *Zookeeper01* > >>> > - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master > > (Including > >>> > Rest)*, *Zookeeper02* > >>> > - Server 3 - *Zookeeper03* > >>> > - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region > > Server.* > >>> > > >>> > The type of operations I am committing on HBase using Zookeeper > > are: > >>> > > >>> > - Simple read/writes/updates using SCAN / MapReduce / Cascading > > .. > >>> > - Queries of the *Rest* service. > >>> > > >>> > > >>> > Thanks a lot !!! > >>> > * *Ronen Itkin > >>> > > >>> > >> > >> > >> > >> -- > >> * > >> Ronen Itkin* > >> Taykey | www.taykey.com > >> > >> > > > > > > -- > > * > > Ronen Itkin* > > Taykey | www.taykey.com > > >
