Thanks @qiang and I have found the root cause. Yes, it's about ReplicationSink and I have found the log, "2014-09-23,14:58:01,736 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: Replicating for table online_miliao_recent".
Then I look at the relevant code and found that every time it calls replicateEntries(), it invokes sharedHBaseAdmin.tableExists(table) as well. The sharedHBaseAdmin.tableExists() will create a new CatalogTracker object which is also a ZooKeeper client. When this method exits, it will cleanup the ZooKeeper client and the session. So this log looks reasonable because the Replication is running. But the tableExists() is a little heavy and I don't think we should invoke it for each time I replicate enties. I also notice that CatalogTracker is not in ReplicationSink after 0.94.11 so it's not a problem for the later versions. It would be great if I have found the jira which removes the CatalogTracker from ReplicationSink :-) On Wed, Sep 24, 2014 at 6:46 PM, Qiang Tian <[email protected]> wrote: > yeah...replication uses client API to process the edits.. > see ReplicationSink#replicateEntries. > perhaps adding a stacktrace here is the best(and fastest) way to find the > root: > > 2014-09-18,16:38:17,867 INFO org.apache.zookeeper.ZooKeeper: Initiating > client connection, connectString=10.2.201.74:11000,10.2.201.73:11000, > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > sessionTimeout=30000 > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@69d892a1 > > > On Wed, Sep 24, 2014 at 10:11 AM, tobe <[email protected]> wrote: > > > It seems to be related to replication. When I disable the replication > from > > the other clusters, the problem doesn't occur again. And I print the > DEBUG > > log to find some abnormal result. I don't think we should get the > response > > of "value=-2209236057317107828" when scan the META table. > > > > 2014-09-23,14:57:59,126 DEBUG > org.apache.hadoop.hbase.client.ClientScanner: > > Creating scanner over .META. starting at key 'online_miliao_message,,' > > 2014-09-23,14:57:59,126 DEBUG > org.apache.hadoop.hbase.client.ClientScanner: > > Advancing internal scanner to startKey at 'online_miliao_message,,' > > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.HBaseClient: IPC > Client > > (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600 from > > hbase_srv/[email protected] sending #208 > > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.SecureClient: IPC > > Client (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600 > > from hbase_srv/[email protected] got value #208 > > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.SecureClient: call > #208 > > state is 0 > > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.SecureClient: call > > #208, response is: > > OW[class=long,value=-2209236057317107828] > > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.SecureRpcEngine: > Call: > > openScanner 0 > > 2014-09-23,14:57:59,126 DEBUG org.apache.hadoop.ipc.HBaseClient: IPC > Client > > (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600 from > > hbase_srv/[email protected] sending #209 > > 2014-09-23,14:57:59,128 DEBUG > > org.apache.hadoop.hbase.security.HBaseSaslRpcServer: SASL server GSSAPI > > callback: setting canonicalized client ID: hbase_srv/[email protected] > > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.hbase.ipc.SecureServer: > > SASL server context established. Negotiated QoP is auth > > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.hbase.ipc.SecureServer: > > SASL server successfully authenticated client: > > hbase_srv/[email protected] (auth:SIMPLE) > > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.ipc.SecureClient: IPC > > Client (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600 > > from hbase_srv/[email protected] got value #209 > > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.ipc.SecureClient: call > #209 > > state is 0 > > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.hbase.ipc.SecureServer: > > Successfully authorized > > > org.apache.hadoop.hbase.ipc.HRegionInterface-hbase_srv/[email protected] > > (auth:SIMPLE) > > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.ipc.SecureClient: call > > #209, response is: > > OW[class=class > > > > > [Lorg.apache.hadoop.hbase.client.Result;,value=[Lorg.apache.hadoop.hbase.client.Result;@6204904d > > ] > > 2014-09-23,14:57:59,128 DEBUG org.apache.hadoop.ipc.SecureRpcEngine: > Call: > > next 2 > > 2014-09-23,14:57:59,129 DEBUG org.apache.hadoop.ipc.HBaseClient: IPC > Client > > (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600 from > > hbase_srv/[email protected] sending #210 > > 2014-09-23,14:57:59,129 DEBUG org.apache.hadoop.ipc.SecureClient: IPC > > Client (847446306) connection to lg-hadoop-srv-st04.bj/10.2.201.77:13600 > > from hbase_srv/[email protected] got value #210 > > 2014-09-23,14:57:59,129 DEBUG org.apache.hadoop.ipc.SecureClient: call > #210 > > state is 0 > > 2014-09-23,14:57:59,129 DEBUG org.apache.hadoop.ipc.SecureClient: call > > #210, response is: > > OW[class=void,value=null] > > 2014-09-23,14:57:59,129 DEBUG org.apache.hadoop.ipc.SecureRpcEngine: > Call: > > close 0 > > > > On Tue, Sep 23, 2014 at 2:27 PM, tobe <[email protected]> wrote: > > > > > I mean I have no idea why the RegionServer has so many log about > > > establishing and closing session. > > > > > > I don't think we have so many clients within every second. This happens > > > when I restart the server but not for all HBase clusters. > > > > > > On Tue, Sep 23, 2014 at 11:57 AM, Qiang Tian <[email protected]> > wrote: > > > > > >> sorry, what reason do you mean? > > >> I took a quick look at 0.94.21, it looks only client code paths call > > >> below > > >> constructor, so this version looks immune to the problem.. > > >> > > >> public CatalogTracker(final Configuration conf) throws IOException { > > >> this(null, conf, null); > > >> } > > >> > > >> On Mon, Sep 22, 2014 at 7:41 PM, tobe <[email protected]> wrote: > > >> > > >> > @qiang I have read about this issue. Is it > > >> > https://issues.apache.org/jira/browse/HBASE-4495? > > >> > > > >> > I look deep in to code and can't find the reason of this. Any > > >> suggestion is > > >> > welcome. > > >> > > > >> > On Mon, Sep 22, 2014 at 6:22 PM, Qiang Tian <[email protected]> > > wrote: > > >> > > > >> > > it looks coming from > > >> > > > > >> > > > > >> > > > >> > > > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hbase/hbase/0.94.19/org/apache/hadoop/hbase/catalog/CatalogTracker.java#186 > > >> > > > > >> > > if no zookeeper watcher passed in, we will connect to zk...this > > logic > > >> > > mostly used in client side(getCatalogTracker), so it looks some > > server > > >> > side > > >> > > code use it as well.. I recall Mikhail has a jira to remove > catalog > > >> > > tracker(forgot jira number...) > > >> > > > > >> > > > > >> > > > > >> > > On Sat, Sep 20, 2014 at 7:38 PM, tobe <[email protected]> > > wrote: > > >> > > > > >> > > > Here's the detailed log and you can see it establishes a few > > >> sessions > > >> > for > > >> > > > every seconds. It doesn't happen in other clusters but you can > see > > >> the > > >> > > > similar log in https://issues.apache.org/jira/browse/HBASE-9779 > . > > >> The > > >> > > HBase > > >> > > > version is 0.9.3 so it print ReplicationSink status as INFO and > > the > > >> > later > > >> > > > version will set it as DEBUG. > > >> > > > > > >> > > > 2014-09-20,17:34:50,953 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:50,954 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:50,954 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:50,954 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > lg-com-master02.bj/10.101.10.67:11000 > > . > > >> > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:50,954 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to lg-com-master02.bj/10.101.10.67:11000 > , > > >> > > > initiating > > >> > > > session > > >> > > > 2014-09-20,17:34:50,956 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> lg-com-master02.bj/10.101.10.67:11000 > > >> > , > > >> > > > sessionid = 0x14877bf55307a85, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:50,962 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x14877bf55307a85 closed > > >> > > > 2014-09-20,17:34:50,962 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:50,963 INFO > > >> > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > > >> Total > > >> > > > replicated: 9 > > >> > > > 2014-09-20,17:34:50,984 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:50,985 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:50,985 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:50,985 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > lg-com-master02.bj/10.101.10.67:11000 > > . > > >> > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:50,985 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to lg-com-master02.bj/10.101.10.67:11000 > , > > >> > > > initiating > > >> > > > session > > >> > > > 2014-09-20,17:34:50,987 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> lg-com-master02.bj/10.101.10.67:11000 > > >> > , > > >> > > > sessionid = 0x14877bf55307a87, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:50,993 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x14877bf55307a87 closed > > >> > > > 2014-09-20,17:34:50,993 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:50,995 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:50,995 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:50,995 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:50,995 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > lg-com-master02.bj/10.101.10.67:11000 > > . > > >> > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:50,996 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to lg-com-master02.bj/10.101.10.67:11000 > , > > >> > > > initiating > > >> > > > session > > >> > > > 2014-09-20,17:34:50,997 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> lg-com-master02.bj/10.101.10.67:11000 > > >> > , > > >> > > > sessionid = 0x14877bf55307a88, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,003 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x14877bf55307a88 closed > > >> > > > 2014-09-20,17:34:51,003 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,004 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,004 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,004 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,005 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,005 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,006 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > >> > > , > > >> > > > sessionid = 0x44877d1f0dfc467, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,011 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x44877d1f0dfc467 closed > > >> > > > 2014-09-20,17:34:51,011 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,013 INFO > > >> > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > > >> Total > > >> > > > replicated: 16 > > >> > > > 2014-09-20,17:34:51,048 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,048 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,048 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,048 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,049 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,050 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > >> > > , > > >> > > > sessionid = 0x24878270100902a, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,056 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x24878270100902a closed > > >> > > > 2014-09-20,17:34:51,056 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,058 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,058 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,058 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,058 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > lg-com-master02.bj/10.101.10.67:11000 > > . > > >> > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,059 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to lg-com-master02.bj/10.101.10.67:11000 > , > > >> > > > initiating > > >> > > > session > > >> > > > 2014-09-20,17:34:51,060 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> lg-com-master02.bj/10.101.10.67:11000 > > >> > , > > >> > > > sessionid = 0x14877bf55307a8b, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,066 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x14877bf55307a8b closed > > >> > > > 2014-09-20,17:34:51,066 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,068 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,068 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,068 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,068 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,069 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,070 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > >> > > , > > >> > > > sessionid = 0x34877bf55f10496, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,075 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x34877bf55f10496 closed > > >> > > > 2014-09-20,17:34:51,075 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,076 INFO > > >> > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > > >> Total > > >> > > > replicated: 12 > > >> > > > 2014-09-20,17:34:51,085 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,086 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,086 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,086 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,086 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,087 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > >> > > , > > >> > > > sessionid = 0x44877d1f0dfc469, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,093 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x44877d1f0dfc469 closed > > >> > > > 2014-09-20,17:34:51,093 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,095 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,095 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,095 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,095 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,096 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,097 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > >> > > , > > >> > > > sessionid = 0x34877bf55f10497, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,103 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x34877bf55f10497 closed > > >> > > > 2014-09-20,17:34:51,103 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,105 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,105 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,105 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,106 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,106 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,107 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > >> > > , > > >> > > > sessionid = 0x34877bf55f10498, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,112 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x34877bf55f10498 closed > > >> > > > 2014-09-20,17:34:51,112 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,115 INFO > > >> > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > > >> Total > > >> > > > replicated: 14 > > >> > > > 2014-09-20,17:34:51,168 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,168 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,168 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,168 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,169 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,170 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > >> > > , > > >> > > > sessionid = 0x44877d1f0dfc46d, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,175 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,175 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x44877d1f0dfc46d closed > > >> > > > 2014-09-20,17:34:51,177 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,177 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,177 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,177 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,177 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,179 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > >> > > , > > >> > > > sessionid = 0x44877d1f0dfc46e, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,184 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x44877d1f0dfc46e closed > > >> > > > 2014-09-20,17:34:51,184 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,185 INFO > > >> > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > > >> Total > > >> > > > replicated: 2 > > >> > > > 2014-09-20,17:34:51,203 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,204 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,204 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,204 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,204 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,205 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct02.bj/10.2.201.74:11000 > > >> > > , > > >> > > > sessionid = 0x34877bf55f1049c, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,211 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x34877bf55f1049c closed > > >> > > > 2014-09-20,17:34:51,211 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,213 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,213 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,213 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,213 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,214 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,215 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > >> > > , > > >> > > > sessionid = 0x24878270100902c, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,221 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x24878270100902c closed > > >> > > > 2014-09-20,17:34:51,221 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,224 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,224 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,224 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,224 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > >> . > > >> > > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,225 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > , > > >> > > > initiating session > > >> > > > 2014-09-20,17:34:51,226 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> > lg-hadoop-srv-ct03.bj/10.2.201.75:11000 > > >> > > , > > >> > > > sessionid = 0x44877d1f0dfc46f, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,232 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x44877d1f0dfc46f closed > > >> > > > 2014-09-20,17:34:51,232 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,233 INFO > > >> > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > > >> Total > > >> > > > replicated: 16 > > >> > > > 2014-09-20,17:34:51,362 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,362 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,362 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,363 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > lg-com-master01.bj/10.101.10.66:11000 > > . > > >> > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,363 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to lg-com-master01.bj/10.101.10.66:11000 > , > > >> > > > initiating > > >> > > > session > > >> > > > 2014-09-20,17:34:51,364 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> lg-com-master01.bj/10.101.10.66:11000 > > >> > , > > >> > > > sessionid = 0x4877f561587a09, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,370 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x4877f561587a09 closed > > >> > > > 2014-09-20,17:34:51,370 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,371 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,371 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,372 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,372 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > lg-com-master01.bj/10.101.10.66:11000 > > . > > >> > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,372 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to lg-com-master01.bj/10.101.10.66:11000 > , > > >> > > > initiating > > >> > > > session > > >> > > > 2014-09-20,17:34:51,373 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> lg-com-master01.bj/10.101.10.66:11000 > > >> > , > > >> > > > sessionid = 0x4877f561587a0b, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,380 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x4877f561587a0b closed > > >> > > > 2014-09-20,17:34:51,380 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,382 INFO org.apache.zookeeper.ZooKeeper: > > >> Initiating > > >> > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000, > > >> > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > sessionTimeout=30000 > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@55d4ee7e > > >> > > > 2014-09-20,17:34:51,382 INFO > > >> > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> identifier > > >> > of > > >> > > > this process is [email protected] > > >> > > > 2014-09-20,17:34:51,382 INFO > > >> > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use > > >> GSSAPI > > >> > > as > > >> > > > SASL mechanism. > > >> > > > 2014-09-20,17:34:51,382 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > socket connection to server > lg-com-master01.bj/10.101.10.66:11000 > > . > > >> > Will > > >> > > > attempt to SASL-authenticate using Login Context section > 'Client' > > >> > > > 2014-09-20,17:34:51,383 INFO org.apache.zookeeper.ClientCnxn: > > Socket > > >> > > > connection established to lg-com-master01.bj/10.101.10.66:11000 > , > > >> > > > initiating > > >> > > > session > > >> > > > 2014-09-20,17:34:51,384 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > establishment complete on server > > >> lg-com-master01.bj/10.101.10.66:11000 > > >> > , > > >> > > > sessionid = 0x4877f561587a0c, negotiated timeout = 30000 > > >> > > > 2014-09-20,17:34:51,390 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > 0x4877f561587a0c closed > > >> > > > 2014-09-20,17:34:51,390 INFO org.apache.zookeeper.ClientCnxn: > > >> > EventThread > > >> > > > shut down > > >> > > > 2014-09-20,17:34:51,391 INFO > > >> > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > > >> Total > > >> > > > replicated: 3 > > >> > > > > > >> > > > > > >> > > > On Sat, Sep 20, 2014 at 1:44 PM, lars hofhansl < > [email protected]> > > >> > wrote: > > >> > > > > > >> > > > > Hi, > > >> > > > > > > >> > > > > can you define "frequently"? > > >> > > > > I.e. send a larger snippet of the log. Connecting every few > > >> minutes > > >> > > would > > >> > > > > OK, Multiple times per second would be strange. > > >> > > > > > > >> > > > > -- Lars > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > ________________________________ > > >> > > > > From: tobe <[email protected]> > > >> > > > > To: "[email protected]" <[email protected]> > > >> > > > > Sent: Thursday, September 18, 2014 1:50 AM > > >> > > > > Subject: HBase establishes session with ZooKeeper and close > the > > >> > session > > >> > > > > immediately > > >> > > > > > > >> > > > > > > >> > > > > I have found that our RegionServers connect to the ZooKeeper > > >> > > frequently. > > >> > > > > They seems to constantly establish the session, close it and > > >> > reconnect > > >> > > > the > > >> > > > > ZooKeeper. Here is the log for both server and client sides. I > > >> have > > >> > no > > >> > > > idea > > >> > > > > why this happens and how to deal with it? We're using HBase > > >> 0.94.11 > > >> > and > > >> > > > > ZooKeeper 3.4.4. > > >> > > > > > > >> > > > > The log from HBase RegionServer: > > >> > > > > > > >> > > > > 2014-09-18,16:38:17,867 INFO org.apache.zookeeper.ZooKeeper: > > >> > Initiating > > >> > > > > client connection, connectString=10.2.201.74:11000, > > >> 10.2.201.73:11000 > > >> > , > > >> > > > > 10.101.10.67:11000,10.101.10.66:11000,10.2.201.75:11000 > > >> > > > > sessionTimeout=30000 > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@69d892a1 > > >> > > > > 2014-09-18,16:38:17,868 INFO > > >> > > > > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will > use > > >> > GSSAPI > > >> > > > as > > >> > > > > SASL mechanism. > > >> > > > > 2014-09-18,16:38:17,868 INFO org.apache.zookeeper.ClientCnxn: > > >> Opening > > >> > > > > socket connection to server > > >> lg-hadoop-srv-ct01.bj/10.2.201.73:11000. > > >> > > > Will > > >> > > > > attempt to SASL-authenticate using Login Context section > > 'Client' > > >> > > > > 2014-09-18,16:38:17,868 INFO > > >> > > > > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The > > >> > identifier > > >> > > of > > >> > > > > this process is [email protected] > > >> > > > > 2014-09-18,16:38:17,868 INFO org.apache.zookeeper.ClientCnxn: > > >> Socket > > >> > > > > connection established to > > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > >> , > > >> > > > > initiating session > > >> > > > > 2014-09-18,16:38:17,870 INFO org.apache.zookeeper.ClientCnxn: > > >> Session > > >> > > > > establishment complete on server > > >> > > lg-hadoop-srv-ct01.bj/10.2.201.73:11000 > > >> > > > , > > >> > > > > sessionid = 0x248782700e52b3c, negotiated timeout = 30000 > > >> > > > > 2014-09-18,16:38:17,876 INFO org.apache.zookeeper.ZooKeeper: > > >> Session: > > >> > > > > 0x248782700e52b3c closed > > >> > > > > 2014-09-18,16:38:17,876 INFO org.apache.zookeeper.ClientCnxn: > > >> > > EventThread > > >> > > > > shut down > > >> > > > > 2014-09-18,16:38:17,878 INFO > > >> > > > > > > org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: > > >> > Total > > >> > > > > replicated: 24 > > >> > > > > > > >> > > > > The log from its ZooKeeper server: > > >> > > > > > > >> > > > > 2014-09-18,16:38:17,869 INFO > > >> > > > > org.apache.zookeeper.server.NIOServerCnxnFactory: [myid:2] > > >> Accepted > > >> > > > socket > > >> > > > > connection from /10.2.201.76:55621 > > >> > > > > 2014-09-18,16:38:17,869 INFO > > >> > > org.apache.zookeeper.server.ZooKeeperServer: > > >> > > > > [myid:2] Client attempting to establish new session at / > > >> > > > 10.2.201.76:55621 > > >> > > > > 2014-09-18,16:38:17,870 INFO > > >> > > org.apache.zookeeper.server.ZooKeeperServer: > > >> > > > > [myid:2] Established session 0x248782700e52b3c with negotiated > > >> > timeout > > >> > > > > 30000 for client /10.2.201.76:55621 > > >> > > > > 2014-09-18,16:38:17,872 INFO > > >> > > > > org.apache.zookeeper.server.auth.SaslServerCallbackHandler: > > >> [myid:2] > > >> > > > > Successfully authenticated client: > > >> > > > > authenticationID=hbase_srv/[email protected]; > > >> > > > > authorizationID=hbase_srv/[email protected]. > > >> > > > > 2014-09-18,16:38:17,872 INFO > > >> > > > > org.apache.zookeeper.server.auth.SaslServerCallbackHandler: > > >> [myid:2] > > >> > > > > Setting authorizedID: hbase_srv > > >> > > > > 2014-09-18,16:38:17,872 INFO > > >> > > org.apache.zookeeper.server.ZooKeeperServer: > > >> > > > > [myid:2] adding SASL authorization for authorizationID: > > hbase_srv > > >> > > > > 2014-09-18,16:38:17,877 INFO > > >> > org.apache.zookeeper.server.NIOServerCnxn: > > >> > > > > [myid:2] Closed socket connection for client / > 10.2.201.76:55621 > > >> > which > > >> > > > had > > >> > > > > sessionid 0x248782700e52b3c > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
