Savonitar opened a new pull request, #352:
URL: https://github.com/apache/flink-connector-kafka/pull/352

   ## What is the purpose of the change
   
   Prevent data loss when Kafka sinks use `TransactionNamingStrategy.POOLING` 
with overlapping transactional ID prefixes. Fixes 
[FLINK-41598](https://issues.apache.org/jira/browse/FLINK-41598).
   
   During startup, a writer with prefix `sink` could abort a listed transaction 
such as `sink-2-0-0`, which belongs to a writer with prefix `sink-2`. If that 
transaction was awaiting a checkpoint commit, the abort could fence its 
producer and its records could be lost. A listed ID such as `sink-a-b` could 
also cause startup to fail with `NumberFormatException`.
   
   ## Brief change log
   
   Listing cleanup now checks that a transactional ID has an exact current or 
restored prefix followed by a subtask ID and offset in the format generated by 
`TransactionalIdFactory`. It skips unrelated IDs before checking subtask 
ownership.
   
   ## Verifying this change
   
   - Unit tests cover overlapping, changed, and empty prefixes, malformed IDs, 
and cleanup continuing past a foreign transaction to an owned one.
   - Six Kafka integration cases verify that the foreign producer can commit 
and that a `read_committed` consumer sees its record. With the old filter, four 
cases failed with `ProducerFencedException` and two failed with 
`NumberFormatException`.
   - On JDK 17, `./mvnw -Dflink.forkCountITCase=1 clean verify` passed with 
Flink 2.2.1. All 60 tests in the three affected classes also passed with Flink 
2.3.0.
   
   ## Does this pull request potentially affect one of the following parts?
   
   | Area | Impact |
   | --- | --- |
   | Dependencies | No |
   | Public API, Table options, or PyFlink wrappers | No |
   | Checkpointed state, serializers, or exactly-once delivery | Yes. The fix 
affects transaction cleanup during exactly-once writer startup. Checkpointed 
state and serializers are unchanged. |
   | Per-record code paths | No |
   
   ## Documentation
   
   This is a bug fix. New feature documentation and translations are not 
applicable.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [x] Yes
   
   Generated-by: Claude Code (Claude Fable 5.2)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to