Hi, the problem is gone. I did what you say :) Thanks! 2014-12-13 22:38 GMT+03:00 Serega Sheypak <serega.shey...@gmail.com>:
> Great, I'll refactor the code. and report back > > 2014-12-13 22:36 GMT+03:00 Stack <st...@duboce.net>: >> >> On Sat, Dec 13, 2014 at 11:33 AM, Serega Sheypak < >> serega.shey...@gmail.com> >> wrote: >> > >> > So the idea is >> > 1. instantiate HConnection using HConnectionManager once >> > 2. Create HTable instance for each Servlet.doPost and close after >> operation >> > is done. >> > Is that correct? >> > >> >> >> Yes. >> >> >> >> > >> > Do region locations cached in this case? >> > Are ZK connections to region/ZK reused? >> > >> >> Yes >> >> >> > Can I "harm" put/get data because of Servlet concurrency? Each >> connection >> > could be used in many threads fro different HTable instances for the >> same >> > HBase table? >> > >> > >> Its a bug if the above has issues. >> >> St.Ack >> >> >> >> > >> > 2014-12-13 22:21 GMT+03:00 lars hofhansl <la...@apache.org>: >> > > >> > > Note also that the createConnection part is somewhat expensive >> (creates a >> > > new thread pool for use with Puts, also does a ZK lookup, etc).If >> > possible >> > > create the connection ahead of time and only get/close an HTable per >> > > request/thread. >> > > -- Lars >> > > From: Serega Sheypak <serega.shey...@gmail.com> >> > > To: user <user@hbase.apache.org> >> > > Sent: Friday, December 12, 2014 11:45 AM >> > > Subject: Re: HConnectionManager leaks with zookeeper conection oo >> many >> > > connections from /my.tomcat.server.com - max is 60 >> > > >> > > i have 10K doPost/doGet requests per second. >> > > Servlet is NOT single-threaded. each doPost/doGet >> > > invokes these lines (encapsulated in DAO): >> > > >> > > 16 HConnection connection = >> > > HConnectionManager.createConnection(config); >> > > 17 HTableInterface table = >> > > connection.getTable(TableName.valueOf("table1")); >> > > >> > > and >> > > >> > > 24 } finally { >> > > 25 table.close(); >> > > 26 connection.close(); >> > > 27 } >> > > >> > > I assumed that this static construction >> > > >> > > 16 HConnection connection = >> > > HConnectionManager.createConnection(config); >> > > >> > > correctly handles multi-threaded access somewhere deep inside. >> > > Right now I don't understand what do I do wrong. >> > > >> > > Try to wrap each your request in Runnable to emulate multi-threaded >> > > pressure on ZK. Your code is linear and mine is not, it's concurrent. >> > > >> > > Thanks.... >> > > >> > > >> > > >> > > >> > > >> > > >> > > 2014-12-12 22:28 GMT+03:00 Stack <st...@duboce.net>: >> > > > >> > > > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted >> > your >> > > > code adding in a put for each cycle. I ran loop 1000 times and no >> > > > complaint from zk. >> > > > >> > > > Tell me more (Is servlet doing single-threaded model? A single >> > > > Configuration is being used or new ones are being created per >> servlet >> > > > invocation? >> > > > >> > > > Below is code and output. >> > > > >> > > > (For better perf, cache the connection) >> > > > St.Ack >> > > > >> > > > >> > > > 1 package org.apache.hadoop.hbase; >> > > > 2 >> > > > 3 import java.io.IOException; >> > > > 4 >> > > > 5 import org.apache.hadoop.conf.Configuration; >> > > > 6 import org.apache.hadoop.hbase.client.HConnection; >> > > > 7 import org.apache.hadoop.hbase.client.HConnectionManager; >> > > > 8 import org.apache.hadoop.hbase.client.HTableInterface; >> > > > 9 import org.apache.hadoop.hbase.client.Put; >> > > > 10 import org.apache.hadoop.hbase.util.Bytes; >> > > > 11 >> > > > 12 public class TestConnnection { >> > > > 13 public static void main(String[] args) throws IOException { >> > > > 14 Configuration config = HBaseConfiguration.create(); >> > > > 15 for (int i = 0; i < 1000; i++) { >> > > > 16 HConnection connection = >> > > > HConnectionManager.createConnection(config); >> > > > 17 HTableInterface table = >> > > > connection.getTable(TableName.valueOf("table1")); >> > > > 18 byte [] cf = Bytes.toBytes("t"); >> > > > 19 try { >> > > > 20 byte [] bytes = Bytes.toBytes(i); >> > > > 21 Put p = new Put(bytes); >> > > > 22 p.add(cf, cf, bytes); >> > > > 23 table.put(p); >> > > > 24 } finally { >> > > > 25 table.close(); >> > > > 26 connection.close(); >> > > > 27 } >> > > > 28 System.out.println("i=" + i); >> > > > 29 } >> > > > 30 } >> > > > 31 } >> > > > >> > > > >> > > > .... >> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper: >> > > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper >> > > > ensemble=localhost:2181 >> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating >> > > client >> > > > connection, connectString=localhost:2181 sessionTimeout=90000 >> > > > watcher=hconnection-0x70dfa475, quorum=localhost:2181, >> baseZNode=/hbase >> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)] >> > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/ >> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL >> (unknown >> > > > error) >> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)] >> > > > zookeeper.ClientCnxn: Socket connection established to localhost/ >> > > > 127.0.0.1:2181, initiating session >> > > > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)] >> > > > zookeeper.ClientCnxn: Session establishment complete on server >> > localhost/ >> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout = >> > > 40000 >> > > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner: >> > Finished >> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', >> > > > STARTKEY => '', ENDKEY => ''} >> > > > 2014-12-12 11:26:10,406 INFO [main] >> > > > client.HConnectionManager$HConnectionImplementation: Closing >> zookeeper >> > > > sessionid=0x14a3fe0664b083a >> > > > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session: >> > > > 0x14a3fe0664b083a closed >> > > > 2014-12-12 11:26:10,407 INFO [main-EventThread] >> zookeeper.ClientCnxn: >> > > > EventThread shut down >> > > > i=997 >> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper: >> > > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper >> > > > ensemble=localhost:2181 >> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating >> > > client >> > > > connection, connectString=localhost:2181 sessionTimeout=90000 >> > > > watcher=hconnection-0x22a7106, quorum=localhost:2181, >> baseZNode=/hbase >> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)] >> > > > zookeeper.ClientCnxn: Opening socket connection to server >> > > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate >> using >> > > SASL >> > > > (unknown error) >> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)] >> > > > zookeeper.ClientCnxn: Socket connection established to >> > > > localhost/0:0:0:0:0:0:0:1:2181, initiating session >> > > > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)] >> > > > zookeeper.ClientCnxn: Session establishment complete on server >> > > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b, >> > negotiated >> > > > timeout = 40000 >> > > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner: >> > Finished >> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', >> > > > STARTKEY => '', ENDKEY => ''} >> > > > 2014-12-12 11:26:10,521 INFO [main] >> > > > client.HConnectionManager$HConnectionImplementation: Closing >> zookeeper >> > > > sessionid=0x14a3fe0664b083b >> > > > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session: >> > > > 0x14a3fe0664b083b closed >> > > > 2014-12-12 11:26:10,521 INFO [main-EventThread] >> zookeeper.ClientCnxn: >> > > > EventThread shut down >> > > > i=998 >> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper: >> > > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper >> > > > ensemble=localhost:2181 >> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating >> > > client >> > > > connection, connectString=localhost:2181 sessionTimeout=90000 >> > > > watcher=hconnection-0x3822f407, quorum=localhost:2181, >> baseZNode=/hbase >> > > > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)] >> > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/ >> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL >> (unknown >> > > > error) >> > > > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)] >> > > > zookeeper.ClientCnxn: Socket connection established to localhost/ >> > > > 127.0.0.1:2181, initiating session >> > > > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)] >> > > > zookeeper.ClientCnxn: Session establishment complete on server >> > localhost/ >> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout = >> > > 40000 >> > > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner: >> > Finished >> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', >> > > > STARTKEY => '', ENDKEY => ''} >> > > > 2014-12-12 11:26:10,638 INFO [main] >> > > > client.HConnectionManager$HConnectionImplementation: Closing >> zookeeper >> > > > sessionid=0x14a3fe0664b083c >> > > > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session: >> > > > 0x14a3fe0664b083c closed >> > > > 2014-12-12 11:26:10,639 INFO [main-EventThread] >> zookeeper.ClientCnxn: >> > > > EventThread shut down >> > > > i=999 >> > > > >> > > > >> > > > >> > > > >> > > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak < >> > > serega.shey...@gmail.com >> > > > > >> > > > wrote: >> > > > >> > > > > Hi, I'm using CDH 5.2, 0.98 >> > > > > I don't know how to use it correctly. I've just used this sample: >> > > > > >> > > > > >> > > > >> > > >> > >> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html >> > > > > >> > > > > HConnection connection = >> > HConnectionManager.createConnection(config); >> > > > > HTableInterface table = >> > > > connection.getTable(TableName.valueOf("table1")); >> > > > > try { >> > > > > // Use the table as needed, for a single operation and a single >> > > thread >> > > > > } finally { >> > > > > table.close(); >> > > > > connection.close(); >> > > > > } >> > > > > >> > > > > >> > > > > Functional testing OK, performance fails :(((( >> > > > > >> > > > > >> > > > > >> > > > > 2014-12-12 19:32 GMT+03:00 Stack <st...@duboce.net>: >> > > > > > >> > > > > > Does zk count go up on each request to the servlet? Which >> version >> > of >> > > > > hbase >> > > > > > so can try on this end? Do you have some client-side log? >> Better >> > if >> > > > you >> > > > > > cache the connection rather than make it each time since setup >> is >> > > > costly >> > > > > > but lets fix first problem first. >> > > > > > St.Ack >> > > > > > >> > > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak < >> > > > > serega.shey...@gmail.com> >> > > > > > wrote: >> > > > > > >> > > > > > > Hi, I'm using HConnectionManager from java servlet >> > > > > > > Looks like it's leaking, all my zookeepers complains that >> there >> > is >> > > > too >> > > > > > many >> > > > > > > connections from servlet hosts. >> > > > > > > Typical line from lZK log: >> > > > > > > >> > > > > > > oo many connections from /my.tomcat.server.com - max is 60 >> > > > > > > >> > > > > > > >> > > > > > > Here is code sample >> > > > > > > >> > > > > > > public class BaseConnection { >> > > > > > > >> > > > > > > private static final Logger LOG = >> > > > > > > LoggerFactory.getLogger(BaseConnection.class); >> > > > > > > >> > > > > > > protected void close(HConnection connection){ >> > > > > > > try{ >> > > > > > > if(connection == null){ >> > > > > > > return; >> > > > > > > } >> > > > > > > connection.close(); >> > > > > > > } >> > > > > > > catch (Exception e){ >> > > > > > > LOG.warn("Error while closing HConnection", e); >> > > > > > > } >> > > > > > > } >> > > > > > > >> > > > > > > protected void close(HTableInterface hTable){ >> > > > > > > try{ >> > > > > > > if(hTable == null){ >> > > > > > > return; >> > > > > > > } >> > > > > > > hTable.close(); >> > > > > > > } >> > > > > > > catch (Exception e){ >> > > > > > > LOG.warn("Error while closing HTable", e); >> > > > > > > } >> > > > > > > } >> > > > > > > } >> > > > > > > >> > > > > > > sample PUT code from subclass: >> > > > > > > >> > > > > > > public void put(List<MyBean> entries){ >> > > > > > > HConnection hConnection = null; >> > > > > > > HTableInterface hTable = null; >> > > > > > > try { >> > > > > > > List<Put> puts = new ArrayList<Put>(entries.size()); >> > > > > > > for(MyBean myBean : entries){ >> > > > > > > puts.add(new MyBeanSerDe().createPut(myBean)); >> > > > > > > } >> > > > > > > hConnection = >> > > > > HConnectionManager.createConnection(configuration); >> > > > > > > hTable = hConnection.getTable(NAME_B); >> > > > > > > hTable.put(puts); >> > > > > > > >> > > > > > > }catch (Exception e){ >> > > > > > > LOG.error("Error while doing bulk put", e); >> > > > > > > }finally{ >> > > > > > > close(hTable); >> > > > > > > close(hConnection); >> > > > > > > } >> > > > > > > } >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > > >> > >> >