[GitHub] [pulsar] rfetishkevi opened a new issue #6563: ZooKeeperChildrenCache reloading creates heavy load on Zookeeper

[GitBox]

rfetishkevi opened a new issue #6563: ZooKeeperChildrenCache reloading creates 
heavy load on Zookeeper
URL: https://github.com/apache/pulsar/issues/6563
 
 
   When topics are frequently created/deleted, Zookeeper is appears to be heavy 
loaded, which makes Pulsar cluster unstable: brokers may loose connection to 
Zookeeper, restart themselves, etc.
   Increasing brokers number from 3 to 12 increased duration of operations that 
involve topics creation by 3 times: from 0.5s to 1.5s in our case. Send message 
operations duration has not increased.
   <img width="1243" alt="Скриншот 2020-03-03 18 40 53" 
src="https://user-images.githubusercontent.com/4807772/77056435-72a30380-69e3-11ea-9bec-89a8c190dd7c.png";>
   As you can see on the screenshot, before adding brokers there are also >0 
outstanding request to the zk leader.
   Before Zookeeper restart, there are lots of messages (tens in second) like 
this in the log:
   ```
   Mar 10 18:29:29 pulsar[39841]: 
[pulsar-ordered-OrderedExecutor-5-0-EventThread] 
   INFO  org.apache.pulsar.zookeeper.ZooKeeperChildrenCache - 
   [State:CONNECTED Timeout:30000 sessionid:0x2000739aa380002 local:/xxx 
remoteserver:xxx:2181 lastZxid:4295513491 xid:268976 sent:268977 recv:287904 
queuedpkts:1 pendingresp:3 queuedevents:0] 
   Received ZooKeeper watch event: WatchedEvent state:SyncConnected 
type:NodeChildrenChanged 
path:/managed-ledgers/<tenant_name>/<cluster_name>/<namespace_name>/persistent
   
   Mar 10 18:29:29 pulsar[39841]: [pulsar-ordered-OrderedExecutor-6-0] INFO  
org.apache.pulsar.zookeeper.ZooKeeperChildrenCache - reloadCache called in 
zookeeperChildrenCache for path 
/managed-ledgers/<tenant_name>/<cluster_name>/<namespace_name>/persistent
   ```
   
   **How to reproduce**
   Pulsar 2.4.1.
   Approximately 2000 create/delete topic operations in a minute. 130000 
short-living topics exist on average.
   
   **How to mitigate**
   Restart all Zookeeper instances one by one. After that topic create/delete 
operation duration decreased dramatically, overall performance improved and 
Pulsar cluster stabilized. Number of messages about reloading zk cache is 
decreased significantly.
   On screenshots below, last zk instance was restarted at 1:46 PM. As you can 
see after that there is no excessive requests/ zk cache reloading.
   <img width="1680" alt="Скриншот 2020-03-19 12 54 38" 
src="https://user-images.githubusercontent.com/4807772/77056676-d2011380-69e3-11ea-8a60-3eb53539a697.png";>
   
   <img width="1680" alt="Скриншот 2020-03-19 12 58 18" 
src="https://user-images.githubusercontent.com/4807772/77056946-40de6c80-69e4-11ea-8a33-c7c79d998123.png";>
   
   **Assumption about cause**
   z-nodes like this contain all persistent topic names that belong to the 
particular namespace.
   `/managed-ledgers/<tenant_name>/<cluster_name>/<namespace_name>/persistent`
   
   Initially they are loaded and cached (with watcher set) during clean start 
of Pulsar cluster:
   `NamespaceService#findBrokerServiceUrl`:
   in method `searchForCandidateBroker`:
   ```
       // Schedule the task to pre-load topics
       pulsar.loadNamespaceTopics(bundle);
   ```
   
   Then, each time topic is created/deleted (which happens frequently in our 
case), watcher is triggered, cache is reloaded (by making request to zk) and 
watcher is set again.
   This repeats very often, and creates heavy load on Zookeeper.
   
   However it looks like that children of nodes 
`/managed-ledgers/<tenant_name>/<cluster_name>/<namespace_name>/persistent` 
aren't used that often, which is proved by restarting zookeeper.
   When zookeeper is restarted, zk-sessions are closed and watchers are 
disconnected.
   So cache is not reloaded unless the 
`/managed-ledgers/<tenant_name>/<cluster_name>/<namespace_name>/persistent` 
node is requested again by broker, which is not the case according to logs and 
JFRs.
   
   **Expected behavior**
   Pulsar cluster is stable and shows good performance when topics are 
created/deleted frequently w/o Zookeeper restart.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services