Excellent! Should be quite a bit faster too.
-- Lars
From: Serega Sheypak <[email protected]>
To: user <[email protected]>
Cc: lars hofhansl <[email protected]>
Sent: Monday, December 15, 2014 5:57 AM
Subject: Re: HConnectionManager leaks with zookeeper conection oo many
connections from /my.tomcat.server.com - max is 60
Hi, the problem is gone.
I did what you say :)
Thanks!
2014-12-13 22:38 GMT+03:00 Serega Sheypak <[email protected]>:
> Great, I'll refactor the code. and report back
>
> 2014-12-13 22:36 GMT+03:00 Stack <[email protected]>:
>>
>> On Sat, Dec 13, 2014 at 11:33 AM, Serega Sheypak <
>> [email protected]>
>> wrote:
>> >
>> > So the idea is
>> > 1. instantiate HConnection using HConnectionManager once
>> > 2. Create HTable instance for each Servlet.doPost and close after
>> operation
>> > is done.
>> > Is that correct?
>> >
>>
>>
>> Yes.
>>
>>
>>
>> >
>> > Do region locations cached in this case?
>> > Are ZK connections to region/ZK reused?
>> >
>>
>> Yes
>>
>>
>> > Can I "harm" put/get data because of Servlet concurrency? Each
>> connection
>> > could be used in many threads fro different HTable instances for the
>> same
>> > HBase table?
>> >
>> >
>> Its a bug if the above has issues.
>>
>> St.Ack
>>
>>
>>
>> >
>> > 2014-12-13 22:21 GMT+03:00 lars hofhansl <[email protected]>:
>> > >
>> > > Note also that the createConnection part is somewhat expensive
>> (creates a
>> > > new thread pool for use with Puts, also does a ZK lookup, etc).If
>> > possible
>> > > create the connection ahead of time and only get/close an HTable per
>> > > request/thread.
>> > > -- Lars
>> > > From: Serega Sheypak <[email protected]>
>> > > To: user <[email protected]>
>> > > Sent: Friday, December 12, 2014 11:45 AM
>> > > Subject: Re: HConnectionManager leaks with zookeeper conection oo
>> many
>> > > connections from /my.tomcat.server.com - max is 60
>> > >
>> > > i have 10K doPost/doGet requests per second.
>> > > Servlet is NOT single-threaded. each doPost/doGet
>> > > invokes these lines (encapsulated in DAO):
>> > >
>> > > 16 HConnection connection =
>> > > HConnectionManager.createConnection(config);
>> > > 17 HTableInterface table =
>> > > connection.getTable(TableName.valueOf("table1"));
>> > >
>> > > and
>> > >
>> > > 24 } finally {
>> > > 25 table.close();
>> > > 26 connection.close();
>> > > 27 }
>> > >
>> > > I assumed that this static construction
>> > >
>> > > 16 HConnection connection =
>> > > HConnectionManager.createConnection(config);
>> > >
>> > > correctly handles multi-threaded access somewhere deep inside.
>> > > Right now I don't understand what do I do wrong.
>> > >
>> > > Try to wrap each your request in Runnable to emulate multi-threaded
>> > > pressure on ZK. Your code is linear and mine is not, it's concurrent.
>> > >
>> > > Thanks....
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > 2014-12-12 22:28 GMT+03:00 Stack <[email protected]>:
>> > > >
>> > > > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted
>> > your
>> > > > code adding in a put for each cycle. I ran loop 1000 times and no
>> > > > complaint from zk.
>> > > >
>> > > > Tell me more (Is servlet doing single-threaded model? A single
>> > > > Configuration is being used or new ones are being created per
>> servlet
>> > > > invocation?
>> > > >
>> > > > Below is code and output.
>> > > >
>> > > > (For better perf, cache the connection)
>> > > > St.Ack
>> > > >
>> > > >
>> > > > 1 package org.apache.hadoop.hbase;
>> > > > 2
>> > > > 3 import java.io.IOException;
>> > > > 4
>> > > > 5 import org.apache.hadoop.conf.Configuration;
>> > > > 6 import org.apache.hadoop.hbase.client.HConnection;
>> > > > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
>> > > > 8 import org.apache.hadoop.hbase.client.HTableInterface;
>> > > > 9 import org.apache.hadoop.hbase.client.Put;
>> > > > 10 import org.apache.hadoop.hbase.util.Bytes;
>> > > > 11
>> > > > 12 public class TestConnnection {
>> > > > 13 public static void main(String[] args) throws IOException {
>> > > > 14 Configuration config = HBaseConfiguration.create();
>> > > > 15 for (int i = 0; i < 1000; i++) {
>> > > > 16 HConnection connection =
>> > > > HConnectionManager.createConnection(config);
>> > > > 17 HTableInterface table =
>> > > > connection.getTable(TableName.valueOf("table1"));
>> > > > 18 byte [] cf = Bytes.toBytes("t");
>> > > > 19 try {
>> > > > 20 byte [] bytes = Bytes.toBytes(i);
>> > > > 21 Put p = new Put(bytes);
>> > > > 22 p.add(cf, cf, bytes);
>> > > > 23 table.put(p);
>> > > > 24 } finally {
>> > > > 25 table.close();
>> > > > 26 connection.close();
>> > > > 27 }
>> > > > 28 System.out.println("i=" + i);
>> > > > 29 }
>> > > > 30 }
>> > > > 31 }
>> > > >
>> > > >
>> > > > ....
>> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
>> > > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
>> > > > ensemble=localhost:2181
>> > > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating
>> > > client
>> > > > connection, connectString=localhost:2181 sessionTimeout=90000
>> > > > watcher=hconnection-0x70dfa475, quorum=localhost:2181,
>> baseZNode=/hbase
>> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
>> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL
>> (unknown
>> > > > error)
>> > > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
>> > > > 127.0.0.1:2181, initiating session
>> > > > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Session establishment complete on server
>> > localhost/
>> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
>> > > 40000
>> > > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner:
>> > Finished
>> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
>> > > > STARTKEY => '', ENDKEY => ''}
>> > > > 2014-12-12 11:26:10,406 INFO [main]
>> > > > client.HConnectionManager$HConnectionImplementation: Closing
>> zookeeper
>> > > > sessionid=0x14a3fe0664b083a
>> > > > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
>> > > > 0x14a3fe0664b083a closed
>> > > > 2014-12-12 11:26:10,407 INFO [main-EventThread]
>> zookeeper.ClientCnxn:
>> > > > EventThread shut down
>> > > > i=997
>> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
>> > > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
>> > > > ensemble=localhost:2181
>> > > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating
>> > > client
>> > > > connection, connectString=localhost:2181 sessionTimeout=90000
>> > > > watcher=hconnection-0x22a7106, quorum=localhost:2181,
>> baseZNode=/hbase
>> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Opening socket connection to server
>> > > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate
>> using
>> > > SASL
>> > > > (unknown error)
>> > > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Socket connection established to
>> > > > localhost/0:0:0:0:0:0:0:1:2181, initiating session
>> > > > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Session establishment complete on server
>> > > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b,
>> > negotiated
>> > > > timeout = 40000
>> > > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner:
>> > Finished
>> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
>> > > > STARTKEY => '', ENDKEY => ''}
>> > > > 2014-12-12 11:26:10,521 INFO [main]
>> > > > client.HConnectionManager$HConnectionImplementation: Closing
>> zookeeper
>> > > > sessionid=0x14a3fe0664b083b
>> > > > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
>> > > > 0x14a3fe0664b083b closed
>> > > > 2014-12-12 11:26:10,521 INFO [main-EventThread]
>> zookeeper.ClientCnxn:
>> > > > EventThread shut down
>> > > > i=998
>> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
>> > > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
>> > > > ensemble=localhost:2181
>> > > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating
>> > > client
>> > > > connection, connectString=localhost:2181 sessionTimeout=90000
>> > > > watcher=hconnection-0x3822f407, quorum=localhost:2181,
>> baseZNode=/hbase
>> > > > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
>> > > > 127.0.0.1:2181. Will not attempt to authenticate using SASL
>> (unknown
>> > > > error)
>> > > > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Socket connection established to localhost/
>> > > > 127.0.0.1:2181, initiating session
>> > > > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
>> > > > zookeeper.ClientCnxn: Session establishment complete on server
>> > localhost/
>> > > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
>> > > 40000
>> > > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner:
>> > Finished
>> > > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
>> > > > STARTKEY => '', ENDKEY => ''}
>> > > > 2014-12-12 11:26:10,638 INFO [main]
>> > > > client.HConnectionManager$HConnectionImplementation: Closing
>> zookeeper
>> > > > sessionid=0x14a3fe0664b083c
>> > > > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
>> > > > 0x14a3fe0664b083c closed
>> > > > 2014-12-12 11:26:10,639 INFO [main-EventThread]
>> zookeeper.ClientCnxn:
>> > > > EventThread shut down
>> > > > i=999
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
>> > > [email protected]
>> > > > >
>> > > > wrote:
>> > > >
>> > > > > Hi, I'm using CDH 5.2, 0.98
>> > > > > I don't know how to use it correctly. I've just used this sample:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
>> > > > >
>> > > > > HConnection connection =
>> > HConnectionManager.createConnection(config);
>> > > > > HTableInterface table =
>> > > > connection.getTable(TableName.valueOf("table1"));
>> > > > > try {
>> > > > > // Use the table as needed, for a single operation and a single
>> > > thread
>> > > > > } finally {
>> > > > > table.close();
>> > > > > connection.close();
>> > > > > }
>> > > > >
>> > > > >
>> > > > > Functional testing OK, performance fails :((((
>> > > > >
>> > > > >
>> > > > >
>> > > > > 2014-12-12 19:32 GMT+03:00 Stack <[email protected]>:
>> > > > > >
>> > > > > > Does zk count go up on each request to the servlet? Which
>> version
>> > of
>> > > > > hbase
>> > > > > > so can try on this end? Do you have some client-side log?
>> Better
>> > if
>> > > > you
>> > > > > > cache the connection rather than make it each time since setup
>> is
>> > > > costly
>> > > > > > but lets fix first problem first.
>> > > > > > St.Ack
>> > > > > >
>> > > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
>> > > > > [email protected]>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi, I'm using HConnectionManager from java servlet
>> > > > > > > Looks like it's leaking, all my zookeepers complains that
>> there
>> > is
>> > > > too
>> > > > > > many
>> > > > > > > connections from servlet hosts.
>> > > > > > > Typical line from lZK log:
>> > > > > > >
>> > > > > > > oo many connections from /my.tomcat.server.com - max is 60
>> > > > > > >
>> > > > > > >
>> > > > > > > Here is code sample
>> > > > > > >
>> > > > > > > public class BaseConnection {
>> > > > > > >
>> > > > > > > private static final Logger LOG =
>> > > > > > > LoggerFactory.getLogger(BaseConnection.class);
>> > > > > > >
>> > > > > > > protected void close(HConnection connection){
>> > > > > > > try{
>> > > > > > > if(connection == null){
>> > > > > > > return;
>> > > > > > > }
>> > > > > > > connection.close();
>> > > > > > > }
>> > > > > > > catch (Exception e){
>> > > > > > > LOG.warn("Error while closing HConnection", e);
>> > > > > > > }
>> > > > > > > }
>> > > > > > >
>> > > > > > > protected void close(HTableInterface hTable){
>> > > > > > > try{
>> > > > > > > if(hTable == null){
>> > > > > > > return;
>> > > > > > > }
>> > > > > > > hTable.close();
>> > > > > > > }
>> > > > > > > catch (Exception e){
>> > > > > > > LOG.warn("Error while closing HTable", e);
>> > > > > > > }
>> > > > > > > }
>> > > > > > > }
>> > > > > > >
>> > > > > > > sample PUT code from subclass:
>> > > > > > >
>> > > > > > > public void put(List<MyBean> entries){
>> > > > > > > HConnection hConnection = null;
>> > > > > > > HTableInterface hTable = null;
>> > > > > > > try {
>> > > > > > > List<Put> puts = new ArrayList<Put>(entries.size());
>> > > > > > > for(MyBean myBean : entries){
>> > > > > > > puts.add(new MyBeanSerDe().createPut(myBean));
>> > > > > > > }
>> > > > > > > hConnection =
>> > > > > HConnectionManager.createConnection(configuration);
>> > > > > > > hTable = hConnection.getTable(NAME_B);
>> > > > > > > hTable.put(puts);
>> > > > > > >
>> > > > > > > }catch (Exception e){
>> > > > > > > LOG.error("Error while doing bulk put", e);
>> > > > > > > }finally{
>> > > > > > > close(hTable);
>> > > > > > > close(hConnection);
>> > > > > > > }
>> > > > > > > }
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>>
>
