Ronen: HBASE-4508 has been verified at rocketfuel.com for 5 months. If you apply the patch from HBASE-4508 and build HBase 0.90(.5) yourself, you don't need to restart Rest service nightly.
Cheers On Tue, Oct 18, 2011 at 9:38 AM, Ronen Itkin <[email protected]> wrote: > Andrew, Ted - Thanks a lot. > > I made a simple cron job that restarts the HBase Rest Service every night > (it clears all unwanted rest-zk sessions)., and > I am casually monitoring the number of Zookeeper clients connections using > Nagios (analyzing netstat output on Zk servers). > Thats good enough for now (hopping for a new release). > > Thanks again! > > > On Tue, Oct 18, 2011 at 8:43 AM, Ronen Itkin <[email protected]> wrote: > >> >> http://search-hadoop.com/m/WBann1KzrxY1&subj=Re+rest+leaking+ZooKeeper+connections >> >> ---------- Forwarded message ---------- >> From: Niv Mizrahi <[email protected]> >> Date: Mon, Oct 17, 2011 at 1:38 PM >> Subject: Re: HBase Rest & Zookeeper issue >> To: Ronen Itkin <[email protected]> >> >> >> sound like a bad news >> >> http://search-hadoop.com/m/WBann1KzrxY1&subj=Re+rest+leaking+ZooKeeper+connections >> >> >> On Mon, Oct 17, 2011 at 11:36 AM, Niv Mizrahi <[email protected]> wrote: >> >>> i have read it this morining still checking it >>> will update you on any updates. >>> >>> >>> On Mon, Oct 17, 2011 at 11:08 AM, Ronen Itkin <[email protected]> wrote: >>> >>>> אתה יכול בבקשה להתייחס לסוגיית ה >>>> Client HBase Code >>>> >>>> איך לדעתך כדאי להתקדם? >>>> >>>> >>>> ---------- Forwarded message ---------- >>>> From: Andrew Purtell <[email protected]> >>>> Date: Mon, Oct 17, 2011 at 4:39 AM >>>> Subject: Re: HBase Rest & Zookeeper issue >>>> To: "[email protected]" <[email protected]> >>>> >>>> >>>> It may be sufficient for you to simply use the latest ASF release of >>>> HBase 0.90 instead of the CDH version. (Or use a snapshot from >>>> nightly.cloudera.com ?) The leaking of ZooKeeper connections suggests a >>>> particular problem. I went through the REST code on the head of the 0.90 >>>> branch and did not find it; the code is (re)using a shared >>>> Configuration instance. >>>> >>>> If you have written client HBase java code, you should also audit your >>>> code and insure you are NOT creating a new Configuration object instance >>>> whenever you create a new HTable object, instead share a single instance. >>>> Also, insure you are not creating HTable objects without passing in a >>>> shared >>>> Configuration. >>>> >>>> >>>> Another option is to patch the HBase client, to address the likely >>>> underlying problem.See HBASE-3777 / HBASE-4508. >>>> >>>> https://issues.apache.org/jira/browse/HBASE-3777 >>>> https://issues.apache.org/jira/browse/HBASE-4508 >>>> >>>> Best regards, >>>> >>>> >>>> - Andy >>>> >>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein >>>> (via Tom White) >>>> >>>> >>>> ----- Original Message ----- >>>> > From: Ronen Itkin <[email protected]> >>>> > To: [email protected] >>>> > Cc: >>>> > Sent: Sunday, October 16, 2011 8:32 PM >>>> > Subject: HBase Rest & Zookeeper issue >>>> > >>>> > Hey all! >>>> > >>>> > At first it my seem like a Zookeeper issue, but the thing is that it >>>> is very >>>> > HBase related so I figured to send it to this mailing list as well. >>>> > I am currently using Zookeeper 3.3.3-cdh3u1, installed as a >>>> distributed >>>> > cluster with 3 nodes. >>>> > Though I have installed Zookeeper separately, its only purpose right >>>> now is >>>> > to serve HBase clients (hbase-0.90.3-cdh3u1). >>>> > Seems like everything is working great for a day or two and then out >>>> the >>>> > blue, Zookeeper refuses to serve clients requests, >>>> > alerting for the wide known issue of exceeding the defined maximum >>>> > connections allowed for a client. >>>> > I changed this value to 500 and it seems that it just held on for more >>>> time >>>> > and finally reached the threshold of 500 and then stopped working. >>>> > So, I set it to 1000 and right now I am waiting to see what will >>>> happen. >>>> > >>>> > Meanwhile, by auditing the Zookeeper connections manually it seems >>>> like they >>>> > keep growing in a slow - promising pace towards the threshold of 1000 >>>> > connections. >>>> > I have tried to analyze the connections and it looks like almost all >>>> of the >>>> > connections were established from the HBase Rest service towards >>>> zookeeper. >>>> > >>>> > Can someone think of a reason why it keep growing? Does someone has >>>> any >>>> > other suggestions regarding this issue? >>>> > Are there any known issues regarding this situation, combining Rest >>>> and >>>> > Zookeeper? >>>> > >>>> > If it helps, this is my environment design: >>>> > >>>> > - Server 1 - Hadoop NameNode, JobTracker, *Hbase Master (Including >>>> Rest)*, >>>> > *Zookeeper01* >>>> > - Server 2 - Hadoop SecondaryMaster, HBase *Backup Master >>>> (Including >>>> > Rest)*, *Zookeeper02* >>>> > - Server 3 - *Zookeeper03* >>>> > - Servers 4-6 - Hadoop DataNode,TaskTracker, *HBase Region Server.* >>>> > >>>> > The type of operations I am committing on HBase using Zookeeper are: >>>> > >>>> > - Simple read/writes/updates using SCAN / MapReduce / Cascading .. >>>> > - Queries of the *Rest* service. >>>> > >>>> > >>>> > Thanks a lot !!! >>>> > * *Ronen Itkin >>>> > >>>> >>>> >>>> >>>> -- >>>> * >>>> Ronen Itkin* >>>> Taykey | www.taykey.com >>>> >>>> >>> >>> >>> -- >>> *Niv Mizrahi* >>> Taykey | www.taykey.com >>> >>> >> >> >> -- >> *Niv Mizrahi* >> Taykey | www.taykey.com >> >> >> >> >> -- >> * >> Ronen Itkin* >> Taykey | www.taykey.com >> >> > > > -- > * > Ronen Itkin* > Taykey | www.taykey.com > >
