[
https://issues.apache.org/jira/browse/KAFKA-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124607#comment-18124607
]
Jun Rao commented on KAFKA-21204:
---------------------------------
Claud suggested the following approach to find MAX_TIMESTAMP. If segment level
MAX_TIMESTAMP is within the bound (e.g. HWM or Last Stable Offset), we can just
use that. Otherwise, use the index on the segment containing the bound to find
the MAX_TIMESTAMP like the following.
{code:java}
(ts_k, off_k) = timeIndex.lastEntryWithOffsetBelow(bound); // binary search
on the monotonic offset field
(off_c, pos_c) = offsetIndex.lookup(bound); // last
offset-index checkpoint <= bound
tailMax = scanBatchesForMax(view.slice(pos_c, posOf(bound) - pos_c)); // <
index.interval.bytes
M(bound) = maxByTimestamp((ts_k, off_k), tailMax); {code}
> ListOffset may return -1 when offset is available
> -------------------------------------------------
>
> Key: KAFKA-21204
> URL: https://issues.apache.org/jira/browse/KAFKA-21204
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Priority: Major
>
> When serving a ListOffset request, the broker eventually calls FileRecords.
> searchForTimestamp(), which calls batchesFrom(). batchesFrom() bounds the
> iteration by the end of the segment, which could include uncommitted data. In
> ReplicaManger.
> fetchOffset(), we bound the returned offset by lastFetchableOffset. However,
> this means that the returned offset is -1, instead of the actual offset. This
> issue can be more likely surfaced when MAX_TIMESTAMP is used in ListOffset
> and records are appended in increasing timestamp order.
>
> {code:java}
> if (resultHolder.lastFetchableOffset.isPresent &&
> timestampAndOffsetOpt.offset >=
> resultHolder.lastFetchableOffset.get) {
> resultHolder.maybeOffsetsError.map(e => throw e)
> } else {{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
