dybyte opened a new pull request, #23569: URL: https://github.com/apache/kafka/pull/23569
When building the remote log aux state, TierStateMachine truncates the local log, which also clears the leader epoch cache on disk, before it reads the leader epoch checkpoint and the producer snapshot from remote storage. If a remote read fails, the replica claims the remote range [logStartOffset, localLogStartOffset) with no leader epoch entries for it. The replica does not recover from this after a restart or a leader change. The fetcher then starts from the end offset of the truncated log, which is inside the leader's local log, so OFFSET_MOVED_TO_TIERED_STORAGE is never returned again and the aux state is never rebuilt. The replica rejoins the ISR. If it then becomes the leader, it cannot serve the remote range, and its remote log retention deletes the remote segments of the lost epochs as unreferenced. This is the path in the reproduction steps of the JIRA. This change reads both files from remote storage into memory first and modifies the local log only after both reads succeed. A remote failure now leaves the local log unchanged, so the next fetch retries the aux state build. Testing: the new TierStateMachineBuildRemoteLogAuxStateTest makes the remote leader epoch read fail and checks that the log start offsets and the leader epoch cache, in memory and on disk, are unchanged. It fails on trunk and passes with this change. TierStateMachineTest and the tiered storage integration tests AlterLogDirTest, ReassignReplicaMoveAndExpandTest and FetchFromLeaderWithCorruptedCheckpointTest also pass. Co-Authored-By: Claude -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
