The GitHub Actions job "Flink Connector Kafka CI" on 
flink-connector-kafka.git/demo/kafka-epoch-rollover-recovery has failed.
Run started by GitHub user Savonitar (triggered by Savonitar).

Head commit for run:
a822330eccb15b21018803897337b73218140a2c / Aleksandr Savonin 
<[email protected]>
[FLINK-40689][connector/kafka] Fix checkpoint recovery after producer epoch 
rollover

A recovered Kafka producer defaults to transaction protocol V1 even when the
checkpointed transaction used V2. Replaying its commit after V2 advanced the
epoch or rotated the producer ID can therefore fail instead of being recognized
as a retry, repeatedly restarting recovery from the same checkpoint.

Persist the original transaction protocol with the producer ID and epoch and
restore it before EndTxn. Serializer version 2 distinguishes unknown, V1, and 
V2;
version-1 checkpoints remain readable as unknown. Unknown legacy metadata must
not erase V2 discovery on a shared recovery producer, while known V1 
transactions
must still select V1.

Also classify InvalidPidMappingException as a known commit failure, like 
fencing.
After rollover and transactional ID reuse, the coordinator may no longer retain
the old producer mapping, so restoring the same identity cannot repair it. Mark
the committable failed without retrying or failing the task, send the failed
acknowledgement, and dispose of the affected producer through the existing 
failure
path. This applies to both checkpoint recovery and commits on live writer
producers; it does not establish that the transaction was committed.

Legacy recovery without protocol discovery can still reject an already committed
V2 rollover. Handling that rejection as a known failure prevents a restart loop
but can log a false alarm for already visible records. Log invalid mappings at
ERROR for INCREMENTING and WARN for POOLING, explain transactional ID expiration
and rollover/reuse, and document the live-path behavior and possible data loss 
in
both documentation trees.

Cover checkpoint restore, mixed protocols on a shared recovery producer, legacy
state with and without protocol discovery, and real pooled-writer reuse. Require
explicit success or known-failure acknowledgements and committed record
visibility. Validate broker and producer protocol fixtures, share setup helpers,
and omit the redundant V1 boundary-epoch control.

Validation: all 48 focused tests pass, including nine recovery scenarios.
Spotless, Checkstyle, and git diff --check pass.

Report URL: 
https://github.com/apache/flink-connector-kafka/actions/runs/35237913749

With regards,
GitHub Actions via GitBox

Reply via email to