rdhabalia opened a new issue #4276: zookeeper leader log file grows and fills up disk space URL: https://github.com/apache/pulsar/issues/4276 ### Issue When Bk-client deletes the ledger, it also tries to delete ledger's parent nodes blindly and because of that ZK's entire log file fills up with below log `Directory not empty for`. Broker rollovers cursor's metadata-ledger at every few configured hours. Broker with large number of global topics and cursors it generates really huge log (>10GB/day) file at zk-leader which consumes most of the disk space. ``` Tue May 14 00:00:00 2019: 2019-05-14 00:00:00,297 - INFO [ProcessThread(sid:5 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x123454567 type:delete cxid:0xc655f6 zxid:0x123454567 txntype:-1 reqpath:n/a Error Path:/ledgers/000/0000/0123/4567 Error:KeeperErrorCode = Directory not empty for /ledgers/000/0000/0123/4567 ``` ### Possible fix 1. Logging configuration change at ZK/Change log level but then it will prevent us to store important logs when it needs for troubleshooting. 2. Better solution : BK-client should only delete parent node if it's empty. It also prevents unexpected data-loss due to unseen zk's bug while deleting non-empty znode.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
