On Sat, Mar 26, 2011 at 2:08 AM, Dmitriy Lyubimov <[email protected]> wrote:
> or perhaps scratch that. it seems you are saying the problem arises on > the backend and your reducer code certainly doesn't create over 10 > connections there. so it might be a combination of other tasks running > at the same address. > Yes, in my case it's a matter of machines with 20+ MR slots, each of which consumes a ZK connection. -Todd > > On Sat, Mar 26, 2011 at 2:04 AM, Dmitriy Lyubimov <[email protected]> > wrote: > > yes i had a very similar issue although i prefer to think about it in > > terms of hbase and by extension zk connection leak in TableInputFormat > > rather than adjusting max zk connection to 30 'cause sooner or later > > you will run out of it too. > > > > The problem is that HConnectionManager now identifies hbase connection > > by identity comparison of the Configuration object. Which is kind of > > unfortunate because cloning configuration is a common practice and in > > reality it means every MR client would open its own connection to > > hbase. > > > > To exacerbate situation, TableInput/OutputFormats do not relinquish > > unshared connection and there's very little you can do work around > > this. I posted workaround to this i used with cdh3b4 just a day or two > > ago. The issue turned out to be a pain as it really quite hard to > > figure where and whether configuration objects get cloned or not. > > > > Also, if you use HTable, be mindful that you may want to release that > > connection explicitly too by calling > > HConnectionManager.deleteConnection (conf, true). > > > > On Fri, Mar 25, 2011 at 1:57 PM, Alex Baranau <[email protected]> > wrote: > >> I see what you are asking. I'm using stand-alone Zookeeper, not > "internal" > >> one of HBase. So it reads configuration only form zoo.cfg. And it seems > that > >> by default (when maxClientCnxns is absent in it) it acts like > >> maxClientCnxns=10. I'd expect it to be unlimited when this property is > >> omitted. At least it was like this.. > >> > >> Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users > won't > >> like it. What do you guys think? > >> > >> Alex Baranau > >> ---- > >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - > HBase > >> Hadoop ecosystem search :: http://search-hadoop.com/ > >> > >> On Fri, Mar 25, 2011 at 4:00 PM, Stack <[email protected]> wrote: > >> > >>> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau < > [email protected]> > >>> wrote: > >>> > As far as I know HBase configured to initiate up to 30 connections by > >>> > default, and maxClientCnxns for Zookeeper was meant to be 30 as well. > >>> > >>> Yes > >>> > >>> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections > >>> for zk?). Is it possible its not reading hbase-site.xml when the task > >>> runs? > >>> > >>> St.Ack > >>> > >> > > > -- Todd Lipcon Software Engineer, Cloudera
