This is an automated email from the ASF dual-hosted git repository.
thetumbled pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 60eb20d4b99 [improve][pip] PIP-444: Rate limit for deleting ledger to
alleviate the zk pressure. (#24771)
60eb20d4b99 is described below
commit 60eb20d4b998171abde2e3d3750f41d4aeab21c7
Author: Wenzhi Feng <[email protected]>
AuthorDate: Thu Oct 9 18:02:53 2025 +0800
[improve][pip] PIP-444: Rate limit for deleting ledger to alleviate the zk
pressure. (#24771)
Co-authored-by: fengwenzhi <[email protected]>
Co-authored-by: Zixuan Liu <[email protected]>
---
pip/pip-444.md | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 55 insertions(+)
diff --git a/pip/pip-444.md b/pip/pip-444.md
new file mode 100644
index 00000000000..74eb86ae5ec
--- /dev/null
+++ b/pip/pip-444.md
@@ -0,0 +1,55 @@
+
+# PIP-444: Rate limit for deleting ledger to alleviate the zk pressure.
+
+# Background knowledge
+
+Apache Pulsar uses Apache BookKeeper as its storage layer.
+Each topic consists of multiple ledgers, which are created periodically based
on the configuration.
+Each ledger corresponds to a ZooKeeper (ZK) node for metadata storage. When a
ledger is deleted, the corresponding ZK node will also be deleted.
+Retention policies determine how long messages are retained in a topic. When
the retention period is reached, the corresponding ledgers will be deleted.
+
+# Motivation
+
+When the retention of a large topic is reduced, a significant number of
ledgers need to be deleted.
+Since this deletion operation is not rate-limited, it could results in
ZooKeeper (ZK) latency of several minutes,
+which threatens the stability of the entire Pulsar cluster.
+
+# Goals
+
+Add rate limit feature for deleting ledgers.
+
+# Detailed Design
+
+Introduce a new configuration parameter:
+```
+ @FieldContext(
+ category = CATEGORY_STORAGE_ML,
+ doc = "Max number of concurrent requests for deleting ledgers at
broker level"
+ )
+ private int managedLedgerDeleteMaxConcurrentRequests = 1000;
+```
+These parameters can be configured in the `broker.conf` file.
+
+When the `managedLedgerDeleteMaxConcurrentRequests` parameter is set to a
value greater than 0,
+the concurrency of ledger deletions will be limited to the specified value.
+
+# Backward & Forward Compatibility
+
+Fully backward compatible and forward compatible.
+
+## Upgrade
+
+New feature is enabled by default with a rate limit of 1000 deletions per
second and concurrency of 1.
+
+## Downgrade / Rollback
+
+To downgrade or rollback, simply revert the configuration changes in the
`broker.conf` file and restart the broker.
+
+
+# Links
+
+<!--
+Updated afterwards
+-->
+* Mailing List discussion thread:
https://lists.apache.org/thread/ppmvm4noowlrhr229zf65mc44xqhj8k1
+* Mailing List voting thread:
https://lists.apache.org/thread/2dop6kyvrpt6ztz9pvylyl8o2syvh9kw
