Hm... confusing,
So here is the code path:
1. Servlet has doPost method
2. HistoryController instantiated during each doPost request.
HistoryController is Logic wrapper around low-level HBase persistence stuff
Each doPost invokes:
final HistoryController getHistoryController(){
return new HistoryController(configuration: getHBaseConfiguration())
}
getHistoryController().updateHistory(historyEntryVisitorId)
3. HistoryController has method:
private HistoryEntryConnection getConnection(){
new HistoryEntryConnection(configuration)
}
so each
updateHistory(historyEntryVisitorId)
gets new low-level HistoryEntryConnection
instance.
HistoryEntryConnection is dead simple, I've shown it previously
public void put(List<MyBean> entries){
HConnection hConnection = null;
HTableInterface hTable = null;
try {
List<Put> puts = new ArrayList<Put>(entries.size());
for(MyBean myBean : entries){
puts.add(new MyBeanSerDe().createPut(myBean));
}
hConnection = HConnectionManager.createConnection(configuration);
hTable = hConnection.getTable(NAME_B);
hTable.put(puts);
}catch (Exception e){
LOG.error("Error while doing bulk put", e);
}finally{
close(hTable);
close(hConnection);
}
}
Manager, Connection, HTable usage copy-pasted from javadoc. Nothing new.
BTW, is there any possibility to limit quantity of opened connections in
HConnectionManager internals ?
I have several web-apps using HBase as backends. Maybe their cumulative
pressure makes ZK go down...
2014-12-13 4:03 GMT+03:00 Stack <[email protected]>:
>
> On Fri, Dec 12, 2014 at 11:45 AM, Serega Sheypak <[email protected]
> >
> wrote:
> >
> > i have 10K doPost/doGet requests per second.
> >
>
> How many concurrent threads going on in your tomcat? Is the Connection
> shared amongst all threads? Perhaps you have > 60 concurrent connections
> running at a time and each Connection has its own zk instance. Thread
> dump?
>
> Post all the code?
>
> I tried multithreaded with 100 threads but still can't repro (see code
> below). Next would be to get zk to dump concurrent connection count but
> lets see what your answers above are first.
>
> St.Ack
>
>
> public class TestConnnection {
>
> static class Putter extends Thread {
>
> Configuration c;
>
> Putter(final Configuration c, final String i) {
>
> super(i);
>
> this.c = c;
>
> }
>
>
> @Override
>
> public void run() {
>
> try {
>
> for (int i = 0; i < 100; i++) {
>
> HConnection connection = HConnectionManager.createConnection(this.c
> );
>
> HTableInterface table = connection.getTable(TableName.valueOf(
> "table1"));
>
> byte [] cf = Bytes.toBytes("t");
>
> try {
>
> byte [] bytes = Bytes.toBytes(i);
>
> Put p = new Put(bytes);
>
> p.add(cf, cf, bytes);
>
> table.put(p);
>
> } finally {
>
> table.close();
>
> connection.close();
>
> }
>
> System.out.println(getName() + ", i=" + i);
>
> }
>
> } catch (Throwable t) {
>
> throw new RuntimeException(t);
>
> }
>
> }
>
> };
>
>
> public static void main(String[] args) throws IOException {
>
> Configuration config = HBaseConfiguration.create();
>
> for (int i = 0; i < 100; i++) {
>
> Thread t = new Putter(config, "" + i);
>
> t.start();
>
> }
>
> }
>
> }
>
>
>
>
>
> > Servlet is NOT single-threaded. each doPost/doGet
> > invokes these lines (encapsulated in DAO):
> >
> > 16 HConnection connection =
> > HConnectionManager.createConnection(config);
> > 17 HTableInterface table =
> > connection.getTable(TableName.valueOf("table1"));
> >
> > and
> >
> > 24 } finally {
> > 25 table.close();
> > 26 connection.close();
> > 27 }
> >
> > I assumed that this static construction
> >
> > 16 HConnection connection =
> > HConnectionManager.createConnection(config);
> >
> > correctly handles multi-threaded access somewhere deep inside.
> > Right now I don't understand what do I do wrong.
> >
> > Try to wrap each your request in Runnable to emulate multi-threaded
> > pressure on ZK. Your code is linear and mine is not, it's concurrent.
> >
> > Thanks....
> >
> >
> >
> >
> > 2014-12-12 22:28 GMT+03:00 Stack <[email protected]>:
> > >
> > > I cannot reproduce. I stood up a cdh5.2 server and then copy/pasted
> your
> > > code adding in a put for each cycle. I ran loop 1000 times and no
> > > complaint from zk.
> > >
> > > Tell me more (Is servlet doing single-threaded model? A single
> > > Configuration is being used or new ones are being created per servlet
> > > invocation?
> > >
> > > Below is code and output.
> > >
> > > (For better perf, cache the connection)
> > > St.Ack
> > >
> > >
> > > 1 package org.apache.hadoop.hbase;
> > > 2
> > > 3 import java.io.IOException;
> > > 4
> > > 5 import org.apache.hadoop.conf.Configuration;
> > > 6 import org.apache.hadoop.hbase.client.HConnection;
> > > 7 import org.apache.hadoop.hbase.client.HConnectionManager;
> > > 8 import org.apache.hadoop.hbase.client.HTableInterface;
> > > 9 import org.apache.hadoop.hbase.client.Put;
> > > 10 import org.apache.hadoop.hbase.util.Bytes;
> > > 11
> > > 12 public class TestConnnection {
> > > 13 public static void main(String[] args) throws IOException {
> > > 14 Configuration config = HBaseConfiguration.create();
> > > 15 for (int i = 0; i < 1000; i++) {
> > > 16 HConnection connection =
> > > HConnectionManager.createConnection(config);
> > > 17 HTableInterface table =
> > > connection.getTable(TableName.valueOf("table1"));
> > > 18 byte [] cf = Bytes.toBytes("t");
> > > 19 try {
> > > 20 byte [] bytes = Bytes.toBytes(i);
> > > 21 Put p = new Put(bytes);
> > > 22 p.add(cf, cf, bytes);
> > > 23 table.put(p);
> > > 24 } finally {
> > > 25 table.close();
> > > 26 connection.close();
> > > 27 }
> > > 28 System.out.println("i=" + i);
> > > 29 }
> > > 30 }
> > > 31 }
> > >
> > >
> > > ....
> > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.RecoverableZooKeeper:
> > > Process identifier=hconnection-0x70dfa475 connecting to ZooKeeper
> > > ensemble=localhost:2181
> > > 2014-12-12 11:26:10,397 INFO [main] zookeeper.ZooKeeper: Initiating
> > client
> > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > watcher=hconnection-0x70dfa475, quorum=localhost:2181, baseZNode=/hbase
> > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > > error)
> > > 2014-12-12 11:26:10,398 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to localhost/
> > > 127.0.0.1:2181, initiating session
> > > 2014-12-12 11:26:10,401 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> localhost/
> > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083a, negotiated timeout =
> > 40000
> > > 2014-12-12 11:26:10,405 DEBUG [main] client.ClientSmallScanner:
> Finished
> > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > STARTKEY => '', ENDKEY => ''}
> > > 2014-12-12 11:26:10,406 INFO [main]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x14a3fe0664b083a
> > > 2014-12-12 11:26:10,407 INFO [main] zookeeper.ZooKeeper: Session:
> > > 0x14a3fe0664b083a closed
> > > 2014-12-12 11:26:10,407 INFO [main-EventThread] zookeeper.ClientCnxn:
> > > EventThread shut down
> > > i=997
> > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.RecoverableZooKeeper:
> > > Process identifier=hconnection-0x22a7106 connecting to ZooKeeper
> > > ensemble=localhost:2181
> > > 2014-12-12 11:26:10,511 INFO [main] zookeeper.ZooKeeper: Initiating
> > client
> > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > watcher=hconnection-0x22a7106, quorum=localhost:2181, baseZNode=/hbase
> > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server
> > > localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using
> > SASL
> > > (unknown error)
> > > 2014-12-12 11:26:10,512 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to
> > > localhost/0:0:0:0:0:0:0:1:2181, initiating session
> > > 2014-12-12 11:26:10,515 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> > > localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x14a3fe0664b083b,
> negotiated
> > > timeout = 40000
> > > 2014-12-12 11:26:10,519 DEBUG [main] client.ClientSmallScanner:
> Finished
> > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > STARTKEY => '', ENDKEY => ''}
> > > 2014-12-12 11:26:10,521 INFO [main]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x14a3fe0664b083b
> > > 2014-12-12 11:26:10,521 INFO [main] zookeeper.ZooKeeper: Session:
> > > 0x14a3fe0664b083b closed
> > > 2014-12-12 11:26:10,521 INFO [main-EventThread] zookeeper.ClientCnxn:
> > > EventThread shut down
> > > i=998
> > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.RecoverableZooKeeper:
> > > Process identifier=hconnection-0x3822f407 connecting to ZooKeeper
> > > ensemble=localhost:2181
> > > 2014-12-12 11:26:10,627 INFO [main] zookeeper.ZooKeeper: Initiating
> > client
> > > connection, connectString=localhost:2181 sessionTimeout=90000
> > > watcher=hconnection-0x3822f407, quorum=localhost:2181, baseZNode=/hbase
> > > 2014-12-12 11:26:10,628 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Opening socket connection to server localhost/
> > > 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> > > error)
> > > 2014-12-12 11:26:10,629 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Socket connection established to localhost/
> > > 127.0.0.1:2181, initiating session
> > > 2014-12-12 11:26:10,631 INFO [main-SendThread(localhost:2181)]
> > > zookeeper.ClientCnxn: Session establishment complete on server
> localhost/
> > > 127.0.0.1:2181, sessionid = 0x14a3fe0664b083c, negotiated timeout =
> > 40000
> > > 2014-12-12 11:26:10,637 DEBUG [main] client.ClientSmallScanner:
> Finished
> > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > > STARTKEY => '', ENDKEY => ''}
> > > 2014-12-12 11:26:10,638 INFO [main]
> > > client.HConnectionManager$HConnectionImplementation: Closing zookeeper
> > > sessionid=0x14a3fe0664b083c
> > > 2014-12-12 11:26:10,639 INFO [main] zookeeper.ZooKeeper: Session:
> > > 0x14a3fe0664b083c closed
> > > 2014-12-12 11:26:10,639 INFO [main-EventThread] zookeeper.ClientCnxn:
> > > EventThread shut down
> > > i=999
> > >
> > >
> > >
> > >
> > > On Fri, Dec 12, 2014 at 10:22 AM, Serega Sheypak <
> > [email protected]
> > > >
> > > wrote:
> > >
> > > > Hi, I'm using CDH 5.2, 0.98
> > > > I don't know how to use it correctly. I've just used this sample:
> > > >
> > > >
> > >
> >
> https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnectionManager.html
> > > >
> > > > HConnection connection =
> HConnectionManager.createConnection(config);
> > > > HTableInterface table =
> > > connection.getTable(TableName.valueOf("table1"));
> > > > try {
> > > > // Use the table as needed, for a single operation and a single
> > thread
> > > > } finally {
> > > > table.close();
> > > > connection.close();
> > > > }
> > > >
> > > >
> > > > Functional testing OK, performance fails :((((
> > > >
> > > >
> > > >
> > > > 2014-12-12 19:32 GMT+03:00 Stack <[email protected]>:
> > > > >
> > > > > Does zk count go up on each request to the servlet? Which version
> of
> > > > hbase
> > > > > so can try on this end? Do you have some client-side log? Better
> if
> > > you
> > > > > cache the connection rather than make it each time since setup is
> > > costly
> > > > > but lets fix first problem first.
> > > > > St.Ack
> > > > >
> > > > > On Fri, Dec 12, 2014 at 2:47 AM, Serega Sheypak <
> > > > [email protected]>
> > > > > wrote:
> > > > >
> > > > > > Hi, I'm using HConnectionManager from java servlet
> > > > > > Looks like it's leaking, all my zookeepers complains that there
> is
> > > too
> > > > > many
> > > > > > connections from servlet hosts.
> > > > > > Typical line from lZK log:
> > > > > >
> > > > > > oo many connections from /my.tomcat.server.com - max is 60
> > > > > >
> > > > > >
> > > > > > Here is code sample
> > > > > >
> > > > > > public class BaseConnection {
> > > > > >
> > > > > > private static final Logger LOG =
> > > > > > LoggerFactory.getLogger(BaseConnection.class);
> > > > > >
> > > > > > protected void close(HConnection connection){
> > > > > > try{
> > > > > > if(connection == null){
> > > > > > return;
> > > > > > }
> > > > > > connection.close();
> > > > > > }
> > > > > > catch (Exception e){
> > > > > > LOG.warn("Error while closing HConnection", e);
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > protected void close(HTableInterface hTable){
> > > > > > try{
> > > > > > if(hTable == null){
> > > > > > return;
> > > > > > }
> > > > > > hTable.close();
> > > > > > }
> > > > > > catch (Exception e){
> > > > > > LOG.warn("Error while closing HTable", e);
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > sample PUT code from subclass:
> > > > > >
> > > > > > public void put(List<MyBean> entries){
> > > > > > HConnection hConnection = null;
> > > > > > HTableInterface hTable = null;
> > > > > > try {
> > > > > > List<Put> puts = new ArrayList<Put>(entries.size());
> > > > > > for(MyBean myBean : entries){
> > > > > > puts.add(new MyBeanSerDe().createPut(myBean));
> > > > > > }
> > > > > > hConnection =
> > > > HConnectionManager.createConnection(configuration);
> > > > > > hTable = hConnection.getTable(NAME_B);
> > > > > > hTable.put(puts);
> > > > > >
> > > > > > }catch (Exception e){
> > > > > > LOG.error("Error while doing bulk put", e);
> > > > > > }finally{
> > > > > > close(hTable);
> > > > > > close(hConnection);
> > > > > > }
> > > > > > }
> > > > > >
> > > > >
> > > >
> > >
> >
>
