or perhaps scratch that. it seems you are saying the problem arises on the backend and your reducer code certainly doesn't create over 10 connections there. so it might be a combination of other tasks running at the same address.
On Sat, Mar 26, 2011 at 2:04 AM, Dmitriy Lyubimov <[email protected]> wrote: > yes i had a very similar issue although i prefer to think about it in > terms of hbase and by extension zk connection leak in TableInputFormat > rather than adjusting max zk connection to 30 'cause sooner or later > you will run out of it too. > > The problem is that HConnectionManager now identifies hbase connection > by identity comparison of the Configuration object. Which is kind of > unfortunate because cloning configuration is a common practice and in > reality it means every MR client would open its own connection to > hbase. > > To exacerbate situation, TableInput/OutputFormats do not relinquish > unshared connection and there's very little you can do work around > this. I posted workaround to this i used with cdh3b4 just a day or two > ago. The issue turned out to be a pain as it really quite hard to > figure where and whether configuration objects get cloned or not. > > Also, if you use HTable, be mindful that you may want to release that > connection explicitly too by calling > HConnectionManager.deleteConnection (conf, true). > > On Fri, Mar 25, 2011 at 1:57 PM, Alex Baranau <[email protected]> > wrote: >> I see what you are asking. I'm using stand-alone Zookeeper, not "internal" >> one of HBase. So it reads configuration only form zoo.cfg. And it seems that >> by default (when maxClientCnxns is absent in it) it acts like >> maxClientCnxns=10. I'd expect it to be unlimited when this property is >> omitted. At least it was like this.. >> >> Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users won't >> like it. What do you guys think? >> >> Alex Baranau >> ---- >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase >> Hadoop ecosystem search :: http://search-hadoop.com/ >> >> On Fri, Mar 25, 2011 at 4:00 PM, Stack <[email protected]> wrote: >> >>> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <[email protected]> >>> wrote: >>> > As far as I know HBase configured to initiate up to 30 connections by >>> > default, and maxClientCnxns for Zookeeper was meant to be 30 as well. >>> >>> Yes >>> >>> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections >>> for zk?). Is it possible its not reading hbase-site.xml when the task >>> runs? >>> >>> St.Ack >>> >> >
