To expand on Cosmin's answer...
I saw elsewhere that Stack suggested upping the number of zookeeper connections
to 1000. This can be set up in you hbase-site.xml file with the parameter
hbase.zookeeper.property.maxClientCnxns, assuming you are using hbase to manage
zookeeper.
Remember that there are rules of thumb for the number of zookeeper nodes you
want:
1 if your cluster size is < 10 nodes
3 if you cluster size is a bit larger
5 if you want to be robust to single node failure when maintaining a zk node
One zk node per rack, with an odd total
You also want to make sure that zk will not contend for hardware with other
processes. So, if you are running M/R jobs over hbase, you may want separate
nodes for zk. At the very least, each zk node should have its own disk, and
perhaps live on its own network links (if you have free ports)
Note: you can use hbase scripts to manage zk even if you don't want to only use
zk for hbase. Given that the hbase configuration process is a bit more
standard than the zk config's, I find it much easier to operate this way. It
does require editing the hbase start and stop scripts to make sure that you
don't turn zk on and off at inappropriate times.
Dave
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>1000</value>
<description>Property from ZooKeeper's config zoo.cfg.
Limit on number of concurrent connections (at the socket level) that a
single client, identified by IP address, may make to a single member of
the ZooKeeper ensemble. Set high to avoid zk connection issues running
standalone and pseudo-distributed.
</description>
</property>
-----Original Message-----
From: Cosmin Lehene [mailto:[email protected]]
Sent: Saturday, June 04, 2011 2:56 AM
To: [email protected]
Cc: [email protected]
Subject: Re: connection loss for /hbase
Hi Geoff,
The default number of connections in ZK is 10 I believe. You don't need 32K for
100 mappers. But it would help to increase that.
Here's my answer for a related problem
http://stackoverflow.com/questions/6153560/hbase-client-connectionloss-for-hbase-error/
I'm not sure how HBase managed ZK works, but I suspect there's a single
instance (somebody correct me if I'm wrong).
There's a "zk dump" link on the HBase master web interface where you can see
the status of all ZK connections to each node.
You should run a ZK cluster as well on the HBase nodes (3 or 5 nodes, make sure
it's not an even number). Here's a set of puppet recipes for Hadoop, HBase,
Zookeeper https://github.com/hstack/puppet as well if you want to automate the
setup.
Cosmin
On Jun 4, 2011, at 3:28 AM, Geoff Hendrey wrote:
> Hi Folks -
>
>
>
> I am experiencing "ConnectionLoss for /hbase" when I run a mapreduce job
> with around 100 mappers and TableInputFormat. I am running a fully
> distributed cluster, letting hbase manage zookeeper. I am running HBase
> 90.1. Is the solution, per this jira,
> https://issues.apache.org/jira/browse/HBASE-3773 , to increase
> hbase.zookeeper.property.maxClientCnxns to 32k?
>
>
>
> This is the stack trace that I get.
>
>
>
> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
>
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementat
> ion.getZooKeeperWatcher(HConnectionManager.java:988)
>
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementat
> ion.setupZookeeperTrackers(HConnectionManager.java:301)
>
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementat
> ion.<init>(HConnectionManager.java:292)
>
> at
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnect
> ionManager.java:155)
>
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:167)
>
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
>
>
>
>
>
> -geoff
>
