Hey, Another update, I had just stopped the Rest Service and the number of zookeeper client connections from that server dropped from 513 to 3.
On Mon, Oct 17, 2011 at 11:57 AM, Ronen Itkin <[email protected]> wrote: > Hey Andrew, > > Thanks for the response, > Please note that the majority of zookeeper client connections are being > made from the Rest service itself and not from any other Java HBase Client, > Moreover, Rest Client address the Rest Service itself and not zookeepeer > (while The rest Service works with zookeeper-->hbase at the background). > So, I guess its not something in our code, but something regarding Rest > and/or ZooKeeper operations. > > To be certain that the connections grow mainly because the Rest Service and > not because of other HBase Java Clients, I will shut it down for 5-6 Hours. > > Can you please aleborate more about using a "shared single instance" ? is > that something configurable for the Rest implementation? > Is there any possibility that a Java HBase client is not using cache for > zookeeper queries? > > Thanks, > > > On Mon, Oct 17, 2011 at 4:39 AM, Andrew Purtell <[email protected]>wrote: > >> It may be sufficient for you to simply use the latest ASF release of HBase >> 0.90 instead of the CDH version. (Or use a snapshot from >> nightly.cloudera.com ?) The leaking of ZooKeeper connections suggests a >> particular problem. I went through the REST code on the head of the 0.90 >> branch and did not find it; the code is (re)using a shared >> Configuration instance. >> >> If you have written client HBase java code, you should also audit your >> code and insure you are NOT creating a new Configuration object instance >> whenever you create a new HTable object, instead share a single instance. >> Also, insure you are not creating HTable objects without passing in a shared >> Configuration. >> >> >> Another option is to patch the HBase client, to address the likely >> underlying problem.See HBASE-3777 / HBASE-4508. >> >> https://issues.apache.org/jira/browse/HBASE-3777 >> https://issues.apache.org/jira/browse/HBASE-4508 >> >> Best regards, >> >> >> - Andy >> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein >> (via Tom White) >> >> >> ----- Original Message ----- >> > From: Ronen Itkin <[email protected]> >> > To: [email protected] >> > Cc: >> > Sent: Sunday, October 16, 2011 8:32 PM >> > Subject: HBase Rest & Zookeeper issue >> > >> > Hey all! >> > >> > At first it my seem like a Zookeeper issue, but the thing is that it is >> very >> > HBase related so I figured to send it to this mailing list as well. >> > I am currently using Zookeeper 3.3.3-cdh3u1, installed as a distributed >> > cluster with 3 nodes. >> > Though I have installed Zookeeper separately, its only purpose right now >> is >> > to serve HBase clients (hbase-0.90.3-cdh3u1). >> > Seems like everything is working great for a day or two and then out the >> > blue, Zookeeper refuses to serve clients requests, >> > alerting for the wide known issue of exceeding the defined maximum >> > connections allowed for a client. >> > I changed this value to 500 and it seems that it just held on for more >> time >> > and finally reached the threshold of 500 and then stopped working. >> > So, I set it to 1000 and right now I am waiting to see what will happen. >> > >> > Meanwhile, by auditing the Zookeeper connections manually it seems like >> they >> > keep growing in a slow - promising pace towards the threshold of 1000 >> > connections. >> > I have tried to analyze the connections and it looks like almost all of >> the >> > connections were established from the HBase Rest service towards >> zookeeper. >> > >> > Can someone think of a reason why it keep growing? Does someone has any >> > other suggestions regarding this issue? >> > Are there any known issues regarding this situation, combining Rest and >> > Zookeeper? >> > >> > If it helps, this is my environment design: >> > >> > - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including >> Rest)*, >> > *Zookeeper01* >> > - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master (Including >> > Rest)*, *Zookeeper02* >> > - Server 3 - *Zookeeper03* >> > - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.* >> > >> > The type of operations I am committing on HBase using Zookeeper are: >> > >> > - Simple read/writes/updates using SCAN / MapReduce / Cascading .. >> > - Queries of the *Rest* service. >> > >> > >> > Thanks a lot !!! >> > * *Ronen Itkin >> > >> > > > > -- > * > Ronen Itkin* > Taykey | www.taykey.com > > -- * Ronen Itkin* Taykey | www.taykey.com
