Hello,
I'm seeing an issue similar to the one described here:
http://webmail.dev411.com/t/hbase/user/149jddw9gh/hbase-establishes-session-with-zookeeper-and-close-the-session-immediately/.
The only difference is that I don't have a replication set up since HBase is
running in a single-node mode on my local machine. Env is:
- hbase-0.98.8-hadoop2
- embedded ZK managed by HBase
- Titan DB running atop of HBase
The problem is that as I can see from logs, connection establishment and
disposal happens many-many times over the time program is running which
dramatically slows down operation of the program itself. I ran debugger several
times and most of the calls seem to originate from
HBaseAdmin#tableExists(TableName) method, where CatalogTracker is cleaned up
which in turn closes ZK connection.
Is this the way it is supposed to work? Can anything be done to establish ZK
connection, keep it an reuse whenever necessary?
Here's a snippet of log (I tried to select relevant info from my point of view
cause there's a lot of it):
[2015-01-29_14:43:22.701] [pool-195-thread-1] TRACE
o.a.hadoop.hbase.zookeeper.ZKUtil - hconnection-0x5849c845 opening connection
to ZooKeeper ensemble=127.0.0.1:2181
[2015-01-29_14:43:22.701] [pool-195-thread-1] INFO
o.a.h.h.z.RecoverableZooKeeper - Process identifier=hconnection-0x5849c845
connecting to ZooKeeper ensemble=127.0.0.1:2181
[2015-01-29_14:43:22.701] [pool-195-thread-1] INFO
org.apache.zookeeper.ZooKeeper - Initiating client connection,
connectString=127.0.0.1:2181 sessionTimeout=90000
watcher=hconnection-0x5849c845, quorum=127.0.0.1:2181, baseZNode=/hbase
...
[2015-01-29_14:43:22.701] [pool-195-thread-1-SendThread(127.0.0.1:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
error)
...
[2015-01-29_14:43:22.702] [pool-195-thread-1-SendThread(127.0.0.1:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
127.0.0.1/127.0.0.1:2181, initiating session
...
[2015-01-29_14:43:22.703] [pool-195-thread-1-SendThread(127.0.0.1:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
127.0.0.1/127.0.0.1:2181, sessionid = 0x14b35473ea3036d, negotiated timeout =
40000
...
[2015-01-29_14:43:22.704] [pool-195-thread-1-EventThread] DEBUG
o.a.h.h.zookeeper.ZooKeeperWatcher - hconnection-0x5849c845-0x14b35473ea3036d
connected
...
[2015-01-29_14:43:22.705] [pool-195-thread-1] TRACE
o.a.hadoop.hbase.zookeeper.ZKUtil - catalogtracker-on-hconnection-0x5849c845
opening connection to ZooKeeper ensemble=127.0.0.1:2181
[2015-01-29_14:43:22.705] [pool-195-thread-1] INFO
o.a.h.h.z.RecoverableZooKeeper - Process
identifier=catalogtracker-on-hconnection-0x5849c845 connecting to ZooKeeper
ensemble=127.0.0.1:2181
[2015-01-29_14:43:22.705] [pool-195-thread-1] INFO
org.apache.zookeeper.ZooKeeper - Initiating client connection,
connectString=127.0.0.1:2181 sessionTimeout=90000
watcher=catalogtracker-on-hconnection-0x5849c845, quorum=127.0.0.1:2181,
baseZNode=/hbase
...
[2015-01-29_14:43:22.708] [pool-195-thread-1-SendThread(127.0.0.1:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
127.0.0.1/127.0.0.1:2181, sessionid = 0x14b35473ea3036e, negotiated timeout =
40000
...
[2015-01-29_14:43:22.713] [pool-195-thread-1] TRACE
o.a.h.hbase.client.ClientScanner - Scan table=hbase:meta, startRow=titan,,
...
[2015-01-29_14:43:22.717] [pool-195-thread-1] TRACE
o.a.h.hbase.client.ClientScanner - Finished {ENCODED => 1588230740, NAME =>
'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
...
[2015-01-29_14:43:22.717] [pool-195-thread-1] DEBUG
o.a.h.hbase.catalog.CatalogTracker - Stopping catalog tracker
org.apache.hadoop.hbase.catalog.CatalogTracker@48c9ae7f
[2015-01-29_14:43:22.717] [pool-195-thread-1] INFO
o.a.h.h.c.HConnectionManager$HConnectionImplementation - Closing zookeeper
sessionid=0x14b35473ea3036d
...
[2015-01-29_14:43:22.821] [pool-195-thread-1] DEBUG
org.apache.zookeeper.ZooKeeper - Closing session: 0x14b35473ea3036e
[2015-01-29_14:43:22.821] [pool-195-thread-1] DEBUG
org.apache.zookeeper.ClientCnxn - Closing client for session: 0x14b35473ea3036e
--
Best regards,
Andrew
