[
https://issues.apache.org/jira/browse/KAFKA-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18122126#comment-18122126
]
Nilesh Kumar commented on KAFKA-21204:
--------------------------------------
[~junrao] I'd like to pick this up if you're not planning to.
One thing to settle first: for MAX_TIMESTAMP, bounding the scan by position is
not enough. The current UnifiedLog path first picks the segment with the
globally greatest maxTimestampSoFar, then looks inside that segment. If that
segment’s max timestamp is beyond the fetchable bound, we could return -1 even
though the correct max timestamp exists in an earlier batch or segment.
So I think the fix needs to compute “max timestamp within the fetchable
prefix,” which changes both segment selection and batch scanning. Also, since
maxPosition is batch-granular and HW/LSO can fall mid-batch, I think the
exclusive bound offset needs to be pushed down too, not just the file position,
so we never return a record past the fetchable offset.
Does that match what you had in mind?
> ListOffset may return -1 when offset is available
> -------------------------------------------------
>
> Key: KAFKA-21204
> URL: https://issues.apache.org/jira/browse/KAFKA-21204
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Priority: Major
>
> When serving a ListOffset request, the broker eventually calls FileRecords.
> searchForTimestamp(), which calls batchesFrom(). batchesFrom() bounds the
> iteration by the end of the segment, which could include uncommitted data. In
> ReplicaManger.
> fetchOffset(), we bound the returned offset by lastFetchableOffset. However,
> this means that the returned offset is -1, instead of the actual offset. This
> issue can be more likely surfaced when MAX_TIMESTAMP is used in ListOffset
> and records are appended in increasing timestamp order.
>
> {code:java}
> if (resultHolder.lastFetchableOffset.isPresent &&
> timestampAndOffsetOpt.offset >=
> resultHolder.lastFetchableOffset.get) {
> resultHolder.maybeOffsetsError.map(e => throw e)
> } else {{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
