[
https://issues.apache.org/jira/browse/FLINK-41603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hao Li reassigned FLINK-41603:
------------------------------
Assignee: Aleksandr Savonin
> Kafka exactly-once sink can lose records after a commit timeout during
> recovery
> -------------------------------------------------------------------------------
>
> Key: FLINK-41603
> URL: https://issues.apache.org/jira/browse/FLINK-41603
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Reporter: Aleksandr Savonin
> Assignee: Aleksandr Savonin
> Priority: Critical
>
> During recovery, the Flink Kafka sink can acknowledge a transaction that
> Kafka never committed. This can cause data loss for read_committed consumers.
> The issue can occur when a committer restores two transactions, A and B, from
> the same checkpoint, for example after scaling down. The following sequence
> reproduces it:
> 1. Flink calls commitTransaction() for A. The call exceeds max.block.ms, so
> Flink schedules a retry. Kafka can still complete the commit in the
> background.
> 2. Kafka completes A before Flink processes B.
> 3. Flink reuses the same producer for B. The Kafka client returns A’s pending
> success result, and Flink marks B as committed without sending a commit
> request for B.
> 4. Flink retries A. B has already been marked complete, so it is never
> retried. Kafka eventually aborts B when its transaction expires.
> I reproduced this with Kafka 4.2.0. Flink reported success for both
> transactions, but Kafka showed that B was aborted. A read_uncommitted
> consumer saw B’s record, while a read_committed consumer saw zero records.
> Expected behavior: Flink should acknowledge each transaction only after Kafka
> confirms the commit for that specific transaction. If a commit times out, its
> retry should use the producer that issued it, including when several restored
> transactions are pending.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)