[
https://issues.apache.org/jira/browse/FLINK-40689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan reassigned FLINK-40689:
-----------------------------------------
Assignee: Aleksandr Savonin
> Kafka sink checkpoint recovery fails after producer epoch rollover with
> transaction V2
> --------------------------------------------------------------------------------------
>
> Key: FLINK-40689
> URL: https://issues.apache.org/jira/browse/FLINK-40689
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Reporter: Aleksandr Savonin
> Assignee: Aleksandr Savonin
> Priority: Major
>
> Kafka sink checkpoint recovery can repeatedly fail with
> InvalidPidMappingException after a Kafka V2 transaction completes at producer
> epoch 32766.
> The failure occurs when:
> 1. Flink checkpoints a pending committable containing the producer ID and
> epoch.
> 2. Kafka commits the transaction and rotates the producer ID, resetting the
> epoch to zero.
> 3. Flink restores the checkpoint and retries the commit using the saved
> identity.
> The connector restores the producer ID and epoch but does not restore the
> transaction protocol. A fresh recovery producer therefore constructs an
> old-protocol EndTxn request. Kafka rejects the previous producer ID instead
> of applying V2’s idempotent rollover retry handling.
> Recovery fails, and subsequent restarts can restore the same checkpoint and
> fail again. The reproduced transaction’s records are already committed and
> the observed impact is job availability.
> Expected behavior: Recovery should successfully acknowledge an
> already-committed transaction, including after producer ID rollover.
> Reproduction: Verified with Flink 2.2.1, Kafka client 4.2.0, and Kafka broker
> 4.2.0. The integration test advances actual broker epochs, verifies committed
> record visibility, and restores the same checkpoint into three fresh
> committer harnesses. Kafka 3.9 provides the V1 control.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
