[ 
https://issues.apache.org/jira/browse/FLINK-41603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hao Li reassigned FLINK-41603:
------------------------------

    Assignee: Aleksandr Savonin

> Kafka exactly-once sink can lose records after a commit timeout during 
> recovery
> -------------------------------------------------------------------------------
>
>                 Key: FLINK-41603
>                 URL: https://issues.apache.org/jira/browse/FLINK-41603
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kafka
>            Reporter: Aleksandr Savonin
>            Assignee: Aleksandr Savonin
>            Priority: Critical
>
> During recovery, the Flink Kafka sink can acknowledge a transaction that 
> Kafka never committed. This can cause data loss for read_committed consumers.
> The issue can occur when a committer restores two transactions, A and B, from 
> the same checkpoint, for example after scaling down. The following sequence 
> reproduces it:
> 1. Flink calls commitTransaction() for A. The call exceeds max.block.ms, so 
> Flink schedules a retry. Kafka can still complete the commit in the 
> background.
> 2. Kafka completes A before Flink processes B.
> 3. Flink reuses the same producer for B. The Kafka client returns A’s pending 
> success result, and Flink marks B as committed without sending a commit 
> request for B.
> 4. Flink retries A. B has already been marked complete, so it is never 
> retried. Kafka eventually aborts B when its transaction expires.
> I reproduced this with Kafka 4.2.0. Flink reported success for both 
> transactions, but Kafka showed that B was aborted. A read_uncommitted 
> consumer saw B’s record, while a read_committed consumer saw zero records.
> Expected behavior: Flink should acknowledge each transaction only after Kafka 
> confirms the commit for that specific transaction. If a commit times out, its 
> retry should use the producer that issued it, including when several restored 
> transactions are pending.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to