dao-jun commented on code in PR #25873:
URL: https://github.com/apache/pulsar/pull/25873#discussion_r3439574015
##########
pip/pip-480.md:
##########
@@ -0,0 +1,153 @@
+# PIP-480: Add cursorless readEntries API to ManagedLedger
+
+# Background knowledge
+
+Most `ManagedLedger` reads go through `ManagedCursor`.
+A cursor is durable state: read position, mark-delete position, and individual
deleted entries.
+That is the right model for Pulsar subscriptions because the broker owns
subscription progress and uses cursor state for
+acknowledgement, backlog accounting, and ledger retention.
+
+Protocol handlers such as KoP do not always want that state. Kafka offsets are
maintained outside Pulsar's
+`ManagedCursor` abstraction. If KoP opens a cursor only to fetch entries, it
creates extra managed-ledger metadata and
+then has to keep the cursor lifecycle aligned with Kafka offsets.
+
+`ManagedLedger` already exposes `asyncReadEntry(Position, ...)` for reading a
single entry by position, but it does not
+provide a batch read primitive for this use case. Callers have to repeat the
ledger traversal logic themselves: validate
+the start position, cross ledger boundaries, skip empty ledgers, and stop at
the current last confirmed entry.
+
+# Motivation
+
+Provide a cursorless read path for downstream projects like KoP.
Review Comment:
@lhotari
KoP's read path uses non-durable ManagedCursor only because ManagedLedger
lacks a positional batch-read API. Every fetch pops a per-offset cursor from a
cache, calls asyncReadEntries, then immediately fires asyncMarkDelete to undo
the cursor's own backlog-holding side effect.
Cursors are keyed by Kafka offset, churned through three concurrent maps,
with explicit handling for races ("A race - same cursor already cached").
> For the batch read, there's also another problem that this introduces when
there's no "waiting mode". Without a waiting mode, there would be unnecessary
polling when cursors are consuming the tail.
KoP has its own `DelayedFetchPurgatory` which is as same as Kafka, `waiting
mode` handled by it.
> There's currently a reason to have a cursor. For example, a cursor will
prevent trimming while the cursor is active.
For this part, a NonDurableCursor does not prevent the ManagedLedger from
being trimmed; only a DurableCursor does. KoP never creates DurableCursors —
its data cleanup follows the same approach as Kafka.
In summary: for KoP, it only needs a method to read entries from
ManagedLedger, cursor is unnecessary.
> One of the existing problems in the ManagedCursor API is that there's no
streaming API. It would be more useful to start a "reactive" streaming read
with a possibility to cancel. I believe that such an API would be useful for
batching reads
I fully agree with this part, but at this stage, I don't get ready to do it.
It should be a further improvement, maybe I can do it when I available.
I wish the PIP can be approved, after solving our current problems, we can
explore long-term development.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
