nodece commented on code in PR #24771: URL: https://github.com/apache/pulsar/pull/24771#discussion_r2390055658
########## pip/pip-444.md: ########## @@ -0,0 +1,55 @@ + +# PIP-444: Rate limit for deleting ledger to alleviate the zk pressure. + +# Background knowledge + +Apache Pulsar uses Apache BookKeeper as its storage layer. +Each topic consists of multiple ledgers, which are created periodically based on the configuration. +Each ledger corresponds to a ZooKeeper (ZK) node for metadata storage. When a ledger is deleted, the corresponding ZK node will also be deleted. +Retention policies determine how long messages are retained in a topic. When the retention period is reached, the corresponding ledgers will be deleted. + +# Motivation + +When the retention of a large topic is reduced, a significant number of ledgers need to be deleted. +Since this deletion operation is not rate-limited, it could results in ZooKeeper (ZK) latency of several minutes, +which threatens the stability of the entire Pulsar cluster. + +# Goals + +Add rate limit feature for deleting ledgers. + +# Detailed Design + +Introduce a new configuration parameter: +``` + @FieldContext( + category = CATEGORY_STORAGE_ML, + doc = "Max number of concurrent requests for deleting ledgers at broker level" + ) + private int managedLedgerDeleteMaxConcurrentRequests = 1000; +``` +These parameters can be configured in the `broker.conf` file. + +When the `managedLedgerDeleteMaxConcurrentRequests` parameter is set to a value greater than 0, +the concurrency of ledger deletions will be limited to the specified value. + +# Backward & Forward Compatibility + +Fully backward compatible and forward compatible. + +## Upgrade + +New feature is enabled by default with a rate limit of 1000 deletions per second and concurrency of 5. Review Comment: ```suggestion New feature is enabled by default with a rate limit of 1000 deletions per second and concurrency of 1. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
