This is an automated email from the ASF dual-hosted git repository.
daojun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 16556faf41f [improve] [pip] PIP-348: Trigger offload on topic load
stage (#22650)
16556faf41f is described below
commit 16556faf41f803497adae42de66e2e9f139b2b83
Author: Hang Chen <[email protected]>
AuthorDate: Mon May 13 11:30:07 2024 +0800
[improve] [pip] PIP-348: Trigger offload on topic load stage (#22650)
---
pip/pip-348.md | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/pip/pip-348.md b/pip/pip-348.md
new file mode 100644
index 00000000000..7661ef36858
--- /dev/null
+++ b/pip/pip-348.md
@@ -0,0 +1,40 @@
+# PIP-348: Trigger offload on topic load stage
+
+# Background knowledge
+
+Pulsar tiered storage is introduced by
[PIP-17](https://github.com/apache/pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics)
to offload cold data from BookKeeper to external storage. Ledger is the basic
offload unit, and one ledger will trigger offload only when the ledger
rollover. Pulsar topic offload can be triggered by the following ways:
+- Manually trigger offload by using the `bin/pulsar-admin` command.
+- Automatically trigger offload by the offload policy.
+
+
+# Motivation
+For triggering offload, the offload policy is the most common way. The offload
policy can be defined in cluster level, namespace level and topic level, and
the offload policy is triggered by the following ways:
+- One ledger is closed or rollover
+- Check the offload policy
+- Trigger offload if the offload policy is satisfied
+
+If one topic has multiple ledgers and the latest ledgers rollover triggered
offload, all the previous ledgers will be added into pending offload queue and
trigger offload one by one. However, if the topic is unloaded and loaded again,
the offload process will be interrupted and needs to waiting for the next
ledger rollover to trigger offload. This will cause the offload process is not
efficient and the offload process is not triggered in time.
+
+
+# Goals
+
+## In Scope
+
+Trigger offload on topic load stage to improve the offload process efficiency
and make sure the offload process is triggered in time.
+
+
+# Detailed Design
+
+## Design & Implementation Details
+
+When the topic is loaded, we can check the offload policy to see if the
offload policy is satisfied. If the offload policy is satisfied, we can trigger
offload immediately. This will improve the offload process efficiency and make
sure the offload process is triggered in time.
+
+In order to reduce the impact on topic load when Pulsar is upgraded from the
old versions, I introduce a flag named `triggerOffloadOnTopicLoad` to control
whether enable this feature or not.
+
+# Backward & Forward Compatibility
+
+Fully compatible.
+
+# Links
+* Mailing List discussion thread:
https://lists.apache.org/thread/2ndomp8v4wkcykzthhlyjqfmswor88kv
+* Mailing List voting thread:
https://lists.apache.org/thread/q4mfn8x69hbgv19nmqx4dmknl3vsn9y8
