ocadaruma opened a new pull request, #15993: URL: https://github.com/apache/kafka/pull/15993
- [A patch for KAFKA-15046](https://github.com/apache/kafka/pull/14242) got rid of `fsync` on `LeaderEpochFileCache#truncateFromStart/End` for performance reason, but it turned out this could cause corrupted leader-epoch checkpoint file on ungraceful OS shutdown * i.e. OS shuts down in the middle when kernel is writing dirty pages back to the device - To address this problem, this PR makes below changes: * Revert `LeaderEpochCheckpoint#write` to always fsync * `truncateFromStart/End` now call `LeaderEpochCheckpoint#write` asynchronously on scheduler thread - Why we still need to call `LeaderEpochCheckpoint#write`? => To prevent checkpoint file grows indefinitely as leader epoch increments - Why async? => To avoid causing performance issue which #14242 addressed * `UnifiedLog#maybeCreateLeaderEpochCache` now loads epoch entries from checkpoint file only when current cache is absent - This is to prevent potentially-stale (because of async checkpointing mentioned above) checkpoint file is read and causes epoch entries to become incorrect ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
