void-ptr974 commented on PR #25873: URL: https://github.com/apache/pulsar/pull/25873#issuecomment-4880479306
I looked a bit more at the KoP read path and wanted to share a possible direction for discussion, without trying to take over the proposal. My current understanding is that KoP does not really need cursor semantics. It maps Kafka offsets to ManagedLedger positions, creates non-durable cursors mainly to get batch reads, reads with both maxEntries and maxBytes, and then mark-deletes the temporary cursor to avoid backlog side effects. KoP also already has its own delayed fetch mechanism for tail reads. So I wonder whether PIP-480 could be framed as a general cursorless positional reader rather than adding another read method directly to ManagedLedger. The split could be: - ManagedCursor remains for subscription-style consumption: ack state, mark-delete, backlog, retention interaction, cursor lifecycle, and read-or-wait. - A positional reader would be for callers that own progress externally and only need currently readable raw entries by position. The reader would still preserve the important read capabilities that KoP and similar callers need: - caller-provided start position - non-waiting read of currently readable entries - maxEntries limit - maxSizeBytes limit - optional upper bound such as maxPosition - cross-ledger reads - skipping removed empty ledgers / moving to the next valid readable position - raw Entry return, with caller-owned release - explicit handling of empty-at-tail vs unavailable/trimmed start position The main advantage is that these semantics get a dedicated home. `ManagedLedger` would only need a small factory-style method to create the reader, instead of accumulating more read overloads directly. `ManagedCursor` would not need to be refactored in v1 and would keep its current behavior. Implementation-wise, this can stay close to the current PR: - reuse ManagedLedgerImpl ledger-handle lookup - reuse current-ledger visibility through lastConfirmedEntry - reuse entry cache reads - reuse read timeout handling - reuse cross-ledger traversal logic - add a small operation dedicated to cursorless reads - keep it detached from cursor state, so it does not update mark-delete, backlog, waiting cursors, or cursor metadata This makes the implementation scope more controlled: we do not need to change dispatcher/cursor behavior, and future additions such as deadline, cancellation, streaming, or richer result reasons can evolve on the reader interface without further expanding ManagedLedger itself. There is one API question that seems worth discussing early: should the result be only `List<Entry>`, or should it expose enough information to distinguish normal tail/no-data from a start position that has already been trimmed? KoP and other positional callers may need to handle those differently. This direction would keep the KoP use case covered, but it would also be useful for broader Pulsar-side positional reads such as admin/debug reads, read-only inspection, recovery/replay, and scan-style tasks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
