haung921209 opened a new pull request, #23205:
URL: https://github.com/apache/kafka/pull/23205
Moving a partition between log directories replaces the current log with the
future log, but the future log's leader epoch cache is only ever populated
from
the record batches copied into it. Epochs with no records behind them - such
as
the one Partition#makeLeader assigns on every leader election - therefore
exist
only in the current log's cache and are dropped by the swap.
A leader whose cache no longer holds the epoch a follower reports as its last
fetched epoch cannot determine that epoch's end offset, so it answers the
fetch
with OffsetOutOfRangeException. The follower then resets its fetch offset to
the
higher of its own log end offset and the leader's log start offset, which on
an
idle partition is the offset it already had, so nothing is truncated and its
next
fetch is identical. The two loop until records are appended to the partition.
This PR copies the epoch history from the log being replaced. This is sound
because the swap only happens once the future log has caught up, so both logs
hold the same offsets. The parameter guarding the high watermark transfer is
renamed to say why the inheritance is valid.
This appears to be the same defect as KAFKA-15608 (analysis posted on both
tickets). It differs from the earlier attempt in #14553 in two ways: the full
epoch history is carried over rather than only the latest entry (dropping
intermediate record-less epochs would still leave the follower subject to
spurious diverging-epoch truncation), and the transfer is confined to the
caught-up swap path - recoverAbandonedFutureLogs offers no LEO-equality
guarantee, so inheriting there could fabricate epoch entries beyond the log
end.
### Testing
- New Partition test reproduces the reported symptom deterministically on
trunk: after moving an idle partition to another log directory, a follower
fetch with its last fetched epoch fails with
`OffsetOutOfRangeException: Could not determine the end offset of the last
fetched epoch Optional[5]` before this change and is served after it.
- New LogManager test pins the full-history transfer (three epoch entries).
- Targeted suites green: PartitionTest, ReplicaManagerTest, UnifiedLogTest,
LogManagerTest, LeaderEpochFileCacheTest, ReplicaAlterLogDirsThreadTest,
LeaderEpochIntegrationTest, OffsetsForLeaderEpochTest (526 tests, 0
failures).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
