The GitHub Actions job "Flink Connector Kafka CI" on 
flink-connector-kafka.git/demo/kafka-epoch-rollover-recovery has failed.
Run started by GitHub user Savonitar (triggered by MartijnVisser).

Head commit for run:
4f68dd84a9047d8994104c904ab94e17fd631cbe / Aleksandr Savonin 
<[email protected]>
[FLINK-40689][connector/kafka] Fix checkpoint recovery after producer epoch 
rollover

A recovered Kafka producer defaults to transaction protocol V1 even when the
checkpointed transaction used V2. Replaying its commit after V2 advanced the
epoch or rotated the producer ID can therefore fail instead of being recognized
as a retry, repeatedly restarting recovery from the same checkpoint.

Persist the original transaction protocol with the producer ID and epoch and
restore it before EndTxn. Serializer version 2 distinguishes unknown, V1, and 
V2;
version-1 checkpoints remain readable as unknown. Unknown legacy metadata must
not erase V2 discovery on a shared recovery producer, while known V1 
transactions
must still select V1.

Also classify InvalidPidMappingException as a known commit failure, like 
fencing.
After rollover and transactional ID reuse, the coordinator may no longer retain
the old producer mapping, so restoring the same identity cannot repair it. Mark
the committable failed without retrying or failing the task, send the failed
acknowledgement, and dispose of the affected producer through the existing 
failure
path. This applies to both checkpoint recovery and commits on live writer
producers; it does not establish that the transaction was committed.

Legacy recovery without protocol discovery can still reject an already committed
V2 rollover. Handling that rejection as a known failure prevents a restart loop
but can log a false alarm for already visible records. Log invalid mappings at
ERROR for INCREMENTING and WARN for POOLING, explain transactional ID expiration
and rollover/reuse, and document the live-path behavior and possible data loss 
in
English and Chinese.

Cover checkpoint restore, mixed protocols on a shared recovery producer, legacy
state with and without protocol discovery, and real pooled-writer reuse. Require
explicit success or known-failure acknowledgements and committed record
visibility. Validate broker and producer protocol fixtures using only bootstrap
servers for the Admin client, share setup helpers, and omit the redundant V1
boundary-epoch control. Update existing producer and writer integration tests to
restore the protocol from their committables, matching production recovery.

Validation: the 48-test focused suite passed, including nine recovery scenarios.
All 25 tests in the affected producer, writer, and checkpoint recovery classes
also passed after the final cleanup. Spotless, Checkstyle, and git diff --check
pass.

Report URL: 
https://github.com/apache/flink-connector-kafka/actions/runs/35240774276

With regards,
GitHub Actions via GitBox

Reply via email to