Savonitar opened a new pull request, #353: URL: https://github.com/apache/flink-connector-kafka/pull/353
## What is the purpose of the change Fixes [FLINK-41603](https://issues.apache.org/jira/browse/FLINK-41603). When `commitTransaction()` times out, the Kafka producer can still receive the result from the broker. During recovery, if Flink reuses that producer to commit transaction B while the result for transaction A is pending, Kafka can return A’s success. Flink then marks B as committed and drops it from the retry set. If Kafka later aborts B, its records are lost to `read_committed` consumers. ## Brief change log - Preserve the transaction state in `FlinkKafkaInternalProducer` after a commit timeout so a chained writer’s producer can retry the pending commit. - In `KafkaCommitter`, retain the producer for each restored transaction whose commit timed out. Retry that transaction on its original producer and close any retained producers when the committer shuts down. ## Verifying this change - Reproduced the data loss on Kafka 4.2.0. Before the fix, Flink acknowledged both transactions, but Kafka aborted B and a `read_committed` consumer saw none of B’s records. With the fix, both transactions committed and B’s record was visible. - Added unit and broker integration tests for batched retries, a delayed fencing error, and reuse of a writer’s producer after a successful retry. The integration tests passed against Apache Kafka 4.2.0 and Confluent Platform 7.9.2. - `./mvnw clean verify` passed on the PR head with 899 tests passed and one source-migration test skipped. The committer unit and integration tests also passed on Flink 2.3.0. ## Does this pull request potentially affect one of the following parts: - Dependencies — No. - Public API — No. - Checkpointed state, its serializers, or exactly-once delivery — Yes. This fixes exactly-once delivery during commit retries. The change is confined to in-memory producer handling. - Per-record code paths — No. ## Documentation - Does this pull request introduce a new feature? No. - If yes, how is the feature documented? Not applicable. - If the docs changed, are both `docs/content` and `docs/content.zh` updated? Not applicable. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes Generated-by: Claude Code (Claude Fable 5.2) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
