Savonitar opened a new pull request, #353:
URL: https://github.com/apache/flink-connector-kafka/pull/353

   ## What is the purpose of the change
   
   Fixes [FLINK-41603](https://issues.apache.org/jira/browse/FLINK-41603).
   
   When `commitTransaction()` times out, the Kafka producer can still receive 
the result from the broker. During recovery, if Flink reuses that producer to 
commit transaction B while the result for transaction A is pending, Kafka can 
return A’s success. Flink then marks B as committed and drops it from the retry 
set. If Kafka later aborts B, its records are lost to `read_committed` 
consumers.
   
   ## Brief change log
   
   - Preserve the transaction state in `FlinkKafkaInternalProducer` after a 
commit timeout so a chained writer’s producer can retry the pending commit.
   - In `KafkaCommitter`, retain the producer for each restored transaction 
whose commit timed out. Retry that transaction on its original producer and 
close any retained producers when the committer shuts down.
   
   ## Verifying this change
   
   - Reproduced the data loss on Kafka 4.2.0. Before the fix, Flink 
acknowledged both transactions, but Kafka aborted B and a `read_committed` 
consumer saw none of B’s records. With the fix, both transactions committed and 
B’s record was visible.
   - Added unit and broker integration tests for batched retries, a delayed 
fencing error, and reuse of a writer’s producer after a successful retry. The 
integration tests passed against Apache Kafka 4.2.0 and Confluent Platform 
7.9.2.
   - `./mvnw clean verify` passed on the PR head with 899 tests passed and one 
source-migration test skipped. The committer unit and integration tests also 
passed on Flink 2.3.0.
   
   ## Does this pull request potentially affect one of the following parts:
   
   - Dependencies — No.
   - Public API — No.
   - Checkpointed state, its serializers, or exactly-once delivery — Yes. This 
fixes exactly-once delivery during commit retries. The change is confined to 
in-memory producer handling.
   - Per-record code paths — No.
   
   ## Documentation
   
   - Does this pull request introduce a new feature? No.
   - If yes, how is the feature documented? Not applicable.
   - If the docs changed, are both `docs/content` and `docs/content.zh` 
updated? Not applicable.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes
   
   Generated-by: Claude Code (Claude Fable 5.2)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to