Yubiao Feng created FLINK-41009:
-----------------------------------
Summary: Use a single Pulsar transaction per checkpoint in the
Pulsar sink
Key: FLINK-41009
URL: https://issues.apache.org/jira/browse/FLINK-41009
Project: Flink
Issue Type: Improvement
Components: Connectors / Pulsar
Affects Versions: 2.0.0
Reporter: Yubiao Feng
{{Currently, the Pulsar sink creates a separate Pulsar transaction for each
target topic within the same Flink checkpoint.}}
{{This weakens checkpoint-level atomicity: when committing a checkpoint, some
topic transactions may be committed successfully while others fail. A later
recovery may then encounter inconsistent transaction states, including
transaction timeout or transaction-not-found errors.}}
{{Pulsar transactions can span multiple topics and namespaces. The sink should
therefore create and reuse one transaction for all records written during a
single checkpoint, regardless of their target topics.
At checkpoint completion, the sink should emit one committable and commit that
single transaction. A new transaction should only be created after the previous
checkpoint's transaction has been handed over for commit.}}
{{}}
{{Benefits:}}
* {{Provide atomic commit semantics across all Pulsar topics written in one
checkpoint. }}
* {{Avoid partial transaction commits within a checkpoint. }}
* {{Reduce transaction creation and commit overhead. }}
* {{Avoid recovery failures caused by inconsistent per-topic transaction
states. }}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
