[
https://issues.apache.org/jira/browse/FLINK-41009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131669#comment-18131669
]
Ferenc Csaky edited comment on FLINK-41009 at 10/26/26, 6:32 PM:
-----------------------------------------------------------------
[{{e26b600}}|https://github.com/apache/flink-connector-pulsar/commit/e26b60037d59474fa7c15b7783953a2a01654d54]
in main
[{{805034a}}|https://github.com/apache/flink-connector-pulsar/commit/805034a97ad2f1515854923b5d7f92d1c6929201]
in main
was (Author: JIRAUSER306586):
[{{e26b600}}|https://github.com/apache/flink-connector-pulsar/commit/e26b60037d59474fa7c15b7783953a2a01654d54]
in main
> Use a single Pulsar transaction per checkpoint in the Pulsar sink
> -----------------------------------------------------------------
>
> Key: FLINK-41009
> URL: https://issues.apache.org/jira/browse/FLINK-41009
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
> Reporter: Yubiao Feng
> Assignee: Yubiao Feng
> Priority: Major
> Labels: pull-request-available
> Fix For: pulsar-5.0.0
>
>
>
> {{Currently, the Pulsar sink creates a separate Pulsar transaction for each
> target topic within the same Flink checkpoint.}}
> {{This weakens checkpoint-level atomicity: when committing a checkpoint, some
> topic transactions may be committed successfully while others fail. A later
> recovery may then encounter inconsistent transaction states, including
> transaction timeout or transaction-not-found errors.}}
> Pulsar transactions can span multiple topics and namespaces. The sink should
> therefore create and reuse one transaction for all records written during a
> single checkpoint, regardless of their target topics.
> At checkpoint completion, the sink should emit one committable and commit
> that single transaction. A new transaction should only be created after the
> previous checkpoint's transaction has been handed over for commit.
> {{Benefits:}}
> * Provide atomic commit semantics across all Pulsar topics written in one
> checkpoint.
> * Avoid partial transaction commits within a checkpoint.
> * Reduce transaction creation and commit overhead.
> * {{Avoid recovery failures caused by inconsistent per-topic transaction
> states. }}
>
>
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
