Andrew, Ted - Thanks a lot. I made a simple cron job that restarts the HBase Rest Service every night (it clears all unwanted rest-zk sessions)., and I am casually monitoring the number of Zookeeper clients connections using Nagios (analyzing netstat output on Zk servers). Thats good enough for now (hopping for a new release).
Thanks again! On Tue, Oct 18, 2011 at 8:43 AM, Ronen Itkin <[email protected]> wrote: > > http://search-hadoop.com/m/WBann1KzrxY1&subj=Re+rest+leaking+ZooKeeper+connections > > ---------- Forwarded message ---------- > From: Niv Mizrahi <[email protected]> > Date: Mon, Oct 17, 2011 at 1:38 PM > Subject: Re: HBase Rest & Zookeeper issue > To: Ronen Itkin <[email protected]> > > > sound like a bad news > > http://search-hadoop.com/m/WBann1KzrxY1&subj=Re+rest+leaking+ZooKeeper+connections > > > On Mon, Oct 17, 2011 at 11:36 AM, Niv Mizrahi <[email protected]> wrote: > >> i have read it this morining still checking it >> will update you on any updates. >> >> >> On Mon, Oct 17, 2011 at 11:08 AM, Ronen Itkin <[email protected]> wrote: >> >>> אתה יכול בבקשה להתייחס לסוגיית ה >>> Client HBase Code >>> >>> איך לדעתך כדאי להתקדם? >>> >>> ---------- Forwarded message ---------- >>> From: Andrew Purtell <[email protected]> >>> Date: Mon, Oct 17, 2011 at 4:39 AM >>> Subject: Re: HBase Rest & Zookeeper issue >>> To: "[email protected]" <[email protected]> >>> >>> >>> It may be sufficient for you to simply use the latest ASF release of >>> HBase 0.90 instead of the CDH version. (Or use a snapshot from >>> nightly.cloudera.com ?) The leaking of ZooKeeper connections suggests a >>> particular problem. I went through the REST code on the head of the 0.90 >>> branch and did not find it; the code is (re)using a shared >>> Configuration instance. >>> >>> If you have written client HBase java code, you should also audit your >>> code and insure you are NOT creating a new Configuration object instance >>> whenever you create a new HTable object, instead share a single instance. >>> Also, insure you are not creating HTable objects without passing in a shared >>> Configuration. >>> >>> >>> Another option is to patch the HBase client, to address the likely >>> underlying problem.See HBASE-3777 / HBASE-4508. >>> >>> https://issues.apache.org/jira/browse/HBASE-3777 >>> https://issues.apache.org/jira/browse/HBASE-4508 >>> >>> Best regards, >>> >>> >>> - Andy >>> >>> Problems worthy of attack prove their worth by hitting back. - Piet Hein >>> (via Tom White) >>> >>> >>> ----- Original Message ----- >>> > From: Ronen Itkin <[email protected]> >>> > To: [email protected] >>> > Cc: >>> > Sent: Sunday, October 16, 2011 8:32 PM >>> > Subject: HBase Rest & Zookeeper issue >>> > >>> > Hey all! >>> > >>> > At first it my seem like a Zookeeper issue, but the thing is that it is >>> very >>> > HBase related so I figured to send it to this mailing list as well. >>> > I am currently using Zookeeper 3.3.3-cdh3u1, installed as a distributed >>> > cluster with 3 nodes. >>> > Though I have installed Zookeeper separately, its only purpose right >>> now is >>> > to serve HBase clients (hbase-0.90.3-cdh3u1). >>> > Seems like everything is working great for a day or two and then out >>> the >>> > blue, Zookeeper refuses to serve clients requests, >>> > alerting for the wide known issue of exceeding the defined maximum >>> > connections allowed for a client. >>> > I changed this value to 500 and it seems that it just held on for more >>> time >>> > and finally reached the threshold of 500 and then stopped working. >>> > So, I set it to 1000 and right now I am waiting to see what will >>> happen. >>> > >>> > Meanwhile, by auditing the Zookeeper connections manually it seems like >>> they >>> > keep growing in a slow - promising pace towards the threshold of 1000 >>> > connections. >>> > I have tried to analyze the connections and it looks like almost all of >>> the >>> > connections were established from the HBase Rest service towards >>> zookeeper. >>> > >>> > Can someone think of a reason why it keep growing? Does someone has any >>> > other suggestions regarding this issue? >>> > Are there any known issues regarding this situation, combining Rest and >>> > Zookeeper? >>> > >>> > If it helps, this is my environment design: >>> > >>> > - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including >>> Rest)*, >>> > *Zookeeper01* >>> > - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master (Including >>> > Rest)*, *Zookeeper02* >>> > - Server 3 - *Zookeeper03* >>> > - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.* >>> > >>> > The type of operations I am committing on HBase using Zookeeper are: >>> > >>> > - Simple read/writes/updates using SCAN / MapReduce / Cascading .. >>> > - Queries of the *Rest* service. >>> > >>> > >>> > Thanks a lot !!! >>> > * *Ronen Itkin >>> > >>> >>> >>> >>> -- >>> * >>> Ronen Itkin* >>> Taykey | www.taykey.com >>> >>> >> >> >> -- >> *Niv Mizrahi* >> Taykey | www.taykey.com >> >> > > > -- > *Niv Mizrahi* > Taykey | www.taykey.com > > > > > -- > * > Ronen Itkin* > Taykey | www.taykey.com > > -- * Ronen Itkin* Taykey | www.taykey.com
