allthingssecurity opened a new pull request, #18767: URL: https://github.com/apache/iceberg/pull/18767
Under cooperative rebalancing, `close(revoked)` stops the worker, which discards the records buffered for every partition, including those that stay assigned, and `seekToLastCommittedOffsets` then rewinds only the partitions that have a committed offset. A kept partition that has never committed (a new connector, or a newly added topic or partition) keeps its position, so its discarded records are silently skipped. This remembers the first buffered offset of each partition and seeks a partition without a committed offset to it; partitions with a committed offset behave as before. The seek also no longer returns early when no assigned partition has committed. Test plan: new `TestKafkaUtils` (with `MockConsumer`) and `TestCommitterImpl#testCloseRewindsPartitionsWhoseBufferedRecordsWereDiscarded`. With the old seek behaviour restored, all three fail (an uncommitted partition stays at offset 10 instead of rewinding to 3); with this change they pass. End to end, a harness driving the released 1.12.0 sink classes against an in-JVM fake Kafka lost 10 of 40 records in 5 of 5 runs; with these two classes patched it lost none in 5 of 5, with no duplicate rows. `spotlessCheck`, `checkstyleMain`, `checkstyleTest` and `:iceberg-kafka-connect:iceberg-kafka-connect:test` pass; the `kafka-connect-runtime` integration tests (Docker) were not run. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: fully reviewed - Prompt Summary: Model the Kafka Connect sink's commit protocol in TLA+, reproduce exactly-once violations against released jars, and prepare a minimal fix with a regression test. The fix, the tests and this description are generated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
