[
https://issues.apache.org/jira/browse/KAFKA-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124613#comment-18124613
]
Jun Rao commented on KAFKA-21204:
---------------------------------
Also, the current FileRecords conflates:
# the mutable, LEO-extent physical segment — {{{}append{}}},
{{{}truncateTo{}}}, {{{}readInto(pos){}}}, {{{}batchesFrom(pos){}}},
{{{}searchForTimestamp{}}}, used by storage
internals/recovery/compaction/replication/raft; and
# the immutable, offset-bounded read window produced by {{slice(pos, size)}}
and handed to clients.
Claude suggested the following approach to clean it up.
h2. The design: split the mutable segment from the bounded view, with one
chokepoint
Make the bound a type invariant instead of a field + convention.
* {{LogFileRecords}} (internal physical segment) — owns the channel,
{{{}start{}}}, {{{}end = size = LEO{}}}. Keeps the full/mutating/position API:
{{{}append{}}}, {{{}truncateTo{}}}, {{{}readInto(buf,pos){}}},
{{{}batchesFrom(pos){}}}, {{{}slice{}}}. Visible only to storage internals,
recovery, compaction, tiered upload, raft. Does not implement the client-facing
{{Records}} interface, so it can never be passed where a {{Records}} is
expected.
* {{BoundedRecords}} (immutable, {{{}implements Records{}}}) — the window
returned by {{{}slice(...){}}}. Exposes only whole-extent iteration/transfer
({{{}batches(){}}}, {{{}batchIterator(){}}}, {{{}writeTo{}}},
{{searchForTimestamp, }}{{{}sizeInBytes{}}}), with no arbitrary-position
re-entry, no re-widening, no mutation. This is the only
{{{}FileRecords{}}}-backed {{Records}} that flows into {{FetchDataInfo}} and
responses (consumer, replication, share-fetch — all of which already go through
{{{}read{}}}).
> ListOffset may return -1 when offset is available
> -------------------------------------------------
>
> Key: KAFKA-21204
> URL: https://issues.apache.org/jira/browse/KAFKA-21204
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Priority: Major
>
> When serving a ListOffset request, the broker eventually calls FileRecords.
> searchForTimestamp(), which calls batchesFrom(). batchesFrom() bounds the
> iteration by the end of the segment, which could include uncommitted data. In
> ReplicaManger.
> fetchOffset(), we bound the returned offset by lastFetchableOffset. However,
> this means that the returned offset is -1, instead of the actual offset. This
> issue can be more likely surfaced when MAX_TIMESTAMP is used in ListOffset
> and records are appended in increasing timestamp order.
>
> {code:java}
> if (resultHolder.lastFetchableOffset.isPresent &&
> timestampAndOffsetOpt.offset >=
> resultHolder.lastFetchableOffset.get) {
> resultHolder.maybeOffsetsError.map(e => throw e)
> } else {{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
