This is an automated email from the ASF dual-hosted git repository.
penghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 48d5b9d042a [improve][pip] PIP-286: Make the TopicCompactionService to
support find entry based on publishTime or index (#20867)
48d5b9d042a is described below
commit 48d5b9d042ad4ee00b44d8120f476ac75822c69c
Author: Cong Zhao <[email protected]>
AuthorDate: Tue Sep 26 10:31:51 2023 +0800
[improve][pip] PIP-286: Make the TopicCompactionService to support find
entry based on publishTime or index (#20867)
---
pip/pip-286.md | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 59 insertions(+)
diff --git a/pip/pip-286.md b/pip/pip-286.md
new file mode 100644
index 00000000000..8449f424fba
--- /dev/null
+++ b/pip/pip-286.md
@@ -0,0 +1,59 @@
+# Background knowledge
+
+In https://github.com/apache/pulsar/pull/11139 we support get position based
on timestamp, but it doesn't work well with topic compaction enabled because
the data may have been move to the compacted ledger.
+
+In [PIP-278](https://github.com/apache/pulsar/pull/20624) we introduced the
pluggable topic compaction service to extend the compaction.
+
+# Motivation
+
+In order for `get-message-id` to work well with topic compaction enabled we
need to find the position according to publish time from topic compaction
service,
+but `TopicCompactionService` missing a method that find positions according to
publish time or other metadata, so we should add it.
+
+In addition, this method can also be used to find the position/offset
according to offset/timestamp in the KoP.
+
+# Goals
+
+# High Level Design
+
+We need to add a method to `Topic Compaction Service` that can find the
matching position and other metadata information according to publishTime/index,
+since the `TopicCompactionService` interface already has
`@InterfaceStability.Evolving` annotation, so that we are able to add new
methods directly.
+
+# Detailed Design
+
+Add `findEntryByPublishTime` in the `TopicCompactionService` API.
+
+Add `findEntryByEntryIndex` in the `TopicCompactionService` API.
+
+Implement them in the `PulsarTopicCompactionService` using binary search.
+
+When get messageId by timestamp, find position from topicCompactionService if
we can't find position in the manageLedger.
+
+## Public-facing Changes
+
+ ```java
+ @InterfaceAudience.Public
+ @InterfaceStability.Evolving
+ public interface TopicCompactionService extends AutoCloseable {
+
+ /**
+ * Find the first entry that greater or equal to target publishTime.
+ *
+ * @param publishTime the publish time of entry.
+ * @return the first entry that greater or equal to target publishTime,
this entry can be null.
+ */
+ CompletableFuture<Entry> findEntryByPublishTime(long publishTime);
+
+ /**
+ * Find the first entry that greater or equal to target entryIndex.
+ *
+ * @param entryIndex the index of entry.
+ * @return the first entry that greater or equal to target entryIndex, this
entry can be null.
+ */
+ CompletableFuture<Entry> findEntryByEntryIndex(long entryIndex);
+ }
+ ```
+
+# Links
+
+* Mailing List discussion thread:
https://lists.apache.org/thread/85o3sx6rhohvc370j4r7yd2nb1tx736c
+* Mailing List voting thread:
https://lists.apache.org/thread/q27zg49mpr8otwh29s3sncdcx8ly7ws6
\ No newline at end of file
