Our code has been using new HTable(config, tableName) and this issue started to occur dealing with relatively large data set.
On Tue, Jan 4, 2011 at 7:23 PM, Stack <[email protected]> wrote: > When you make a new HTable, do you do new HTable(tableName) or new > HTable(config, tableName)? If you are using the latter, you still run > into the below? > > St.Ack > > On Tue, Jan 4, 2011 at 6:47 PM, Ted Yu <[email protected]> wrote: > > I increased max connections to 40. > > I still got: > > > > 2011-01-04 21:30:05,701 WARN org.apache.zookeeper.server.NIOServerCnxn: > Too > > many connections from /10.202.50.79 - max is 40 > > 2011-01-04 21:30:06,072 WARN org.apache.zookeeper.server.NIOServerCnxn: > Too > > many connections from /10.202.50.80 - max is 40 > > 2011-01-04 21:30:06,458 WARN org.apache.zookeeper.server.NIOServerCnxn: > Too > > many connections from /10.202.50.80 - max is 40 > > 2011-01-04 21:30:06,944 WARN org.apache.zookeeper.server.NIOServerCnxn: > Too > > many connections from /10.202.50.79 - max is 40 > > 2011-01-04 21:30:07,273 WARN org.apache.zookeeper.server.NIOServerCnxn: > Too > > many connections from /10.202.50.80 - max is 40 > > 2011-01-04 21:30:07,665 WARN org.apache.zookeeper.server.NIOServerCnxn: > Too > > many connections from /10.202.50.79 - max is 40 > > 2011-01-04 21:30:07,876 WARN org.apache.zookeeper.server.NIOServerCnxn: > > EndOfStreamException: Unable to read additional data from client > sessionid > > 0x12d52be9c2b001b, likely client has closed socket > > 2011-01-04 21:30:07,876 INFO org.apache.zookeeper.server.NIOServerCnxn: > > Closed socket connection for client /10.202.50.79:43150 which had > sessionid > > 0x12d52be9c2b001b > > 2011-01-04 21:30:07,876 WARN org.apache.zookeeper.server.NIOServerCnxn: > > EndOfStreamException: Unable to read additional data from client > sessionid > > 0x12d52be9c2b008b, likely client has closed socket > > 2011-01-04 21:30:07,876 INFO org.apache.zookeeper.server.NIOServerCnxn: > > Closed socket connection for client /10.202.50.79:26104 which had > sessionid > > 0x12d52be9c2b008b > > 2011-01-04 21:30:07,876 WARN org.apache.zookeeper.server.NIOServerCnxn: > > EndOfStreamException: Unable to read additional data from client > sessionid > > 0x12d52be9c2b010c, likely client has closed socket > > > > I verified maxClientCnxns of 30 in 0.20.6 where we didn't experience this > > problem. > > > > More comment is welcome. > > > > On Tue, Jan 4, 2011 at 11:47 AM, Ted Yu <[email protected]> wrote: > > > >> So I should be using HTablePool. > >> For 0.20.6, I didn't see ConnectionLossException this often. > >> > >> I wonder if something changed from 0.20.6 to 0.90 > >> > >> On Tue, Jan 4, 2011 at 11:29 AM, Stack <[email protected]> wrote: > >> > >>> Are you passing the same Configuration instance when creating your > >>> HTables? See > >>> > http://people.apache.org/~stack/hbase-0.90.0-candidate-2/docs/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html<http://people.apache.org/%7Estack/hbase-0.90.0-candidate-2/docs/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html> > < > http://people.apache.org/%7Estack/hbase-0.90.0-candidate-2/docs/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html > > > >>> if not. It explains how we figure whether zk client, rpc connections, > >>> etc. are shared or not. > >>> > >>> St.Ack > >>> > >>> On Tue, Jan 4, 2011 at 11:12 AM, Jean-Daniel Cryans < > [email protected]> > >>> wrote: > >>> > It's a zookeeper setting, you cannot have by default more than 30 > >>> > connections from the same IP per ZK peer. > >>> > > >>> > If HBase is starting ZK for you, do change > >>> > hbase.zookeeper.property.maxClientCnxns > >>> > > >>> > J-D > >>> > > >>> > On Tue, Jan 4, 2011 at 11:09 AM, Ted Yu <[email protected]> wrote: > >>> >> Hi, > >>> >> I am using HBase 0.90 and our job fails consistently with the > following > >>> >> exception: > >>> >> > >>> >> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: > >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException: > >>> >> KeeperErrorCode = ConnectionLoss for /hbase > >>> >> at > >>> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:147) > >>> >> at > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1035) > >>> >> ... 19 more > >>> >> Caused by: > >>> org.apache.zookeeper.KeeperException$ConnectionLossException: > >>> >> KeeperErrorCode = ConnectionLoss for /hbase > >>> >> at > >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > >>> >> at > >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > >>> >> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:608) > >>> >> at > >>> > org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902) > >>> >> at > >>> > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133) > >>> >> ... 20 more > >>> >> > >>> >> Zookeeper quorum runs on the same node as NameNode. HMaster is on > >>> another > >>> >> node. Hadoop is cdh3b2. > >>> >> > >>> >> In zookeeper log, I see (10.202.50.79 is the same node where the > >>> exception > >>> >> above happened): > >>> >> > >>> >> 2011-01-04 18:47:40,633 WARN > org.apache.zookeeper.server.NIOServerCnxn: > >>> Too > >>> >> many connections from /10.202.50.79 - max is 30 > >>> >> 2011-01-04 18:47:41,187 WARN > org.apache.zookeeper.server.NIOServerCnxn: > >>> Too > >>> >> many connections from /10.202.50.79 - max is 30 > >>> >> 2011-01-04 18:47:42,375 WARN > org.apache.zookeeper.server.NIOServerCnxn: > >>> Too > >>> >> many connections from /10.202.50.79 - max is 30 > >>> >> 2011-01-04 18:47:42,447 WARN > org.apache.zookeeper.server.NIOServerCnxn: > >>> Too > >>> >> many connections from /10.202.50.79 - max is 30 > >>> >> 2011-01-04 18:47:43,113 WARN > org.apache.zookeeper.server.NIOServerCnxn: > >>> >> EndOfStreamException: Unable to read additional data from client > >>> sessionid > >>> >> 0x12d5220eb970025, likely client has closed socket > >>> >> 2011-01-04 18:47:43,113 INFO > org.apache.zookeeper.server.NIOServerCnxn: > >>> >> Closed socket connection for client /10.202.50.79:37845 which had > >>> sessionid > >>> >> 0x12d5220eb970025 > >>> >> 2011-01-04 18:47:43,113 WARN > org.apache.zookeeper.server.NIOServerCnxn: > >>> >> EndOfStreamException: Unable to read additional data from client > >>> sessionid > >>> >> 0x12d5220eb970087, likely client has closed socket > >>> >> > >>> >> Please advise what parameter I should tune. > >>> >> > >>> >> Thanks > >>> >> > >>> > > >>> > >> > >> > > >
