Aleksandr Savonin created FLINK-40689:
-----------------------------------------
Summary: Kafka sink checkpoint recovery fails after producer epoch
rollover with transaction V2
Key: FLINK-40689
URL: https://issues.apache.org/jira/browse/FLINK-40689
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Reporter: Aleksandr Savonin
Kafka sink checkpoint recovery can repeatedly fail with
InvalidPidMappingException after a Kafka V2 transaction completes at producer
epoch 32766.
The failure occurs when:
1. Flink checkpoints a pending committable containing the producer ID and epoch.
2. Kafka commits the transaction and rotates the producer ID, resetting the
epoch to zero.
3. Flink restores the checkpoint and retries the commit using the saved
identity.
The connector restores the producer ID and epoch but does not restore the
transaction protocol. A fresh recovery producer therefore constructs an
old-protocol EndTxn request. Kafka rejects the previous producer ID instead of
applying V2’s idempotent rollover retry handling.
Recovery fails, and subsequent restarts can restore the same checkpoint and
fail again. The reproduced transaction’s records are already committed and the
observed impact is job availability.
Expected behavior: Recovery should successfully acknowledge an
already-committed transaction, including after producer ID rollover.
Reproduction: Verified with Flink 2.2.1, Kafka client 4.2.0, and Kafka broker
4.2.0. The integration test advances actual broker epochs, verifies committed
record visibility, and restores the same checkpoint into three fresh committer
harnesses. Kafka 3.9 provides the V1 control.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
