peiyu created FLINK-41008:
-----------------------------
Summary: Support multi-table transaction stream load in StarRocks
pipeline connector
Key: FLINK-41008
URL: https://issues.apache.org/jira/browse/FLINK-41008
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Reporter: peiyu
h2. Background
The StarRocks Flink Connector introduced multi-table transaction stream
load in 1.2.15 (upstream PR StarRocks/starrocks-connector-for-apache-flink#487),
with several hardening fixes in 1.2.16. It allows multiple tables written
in the same flush cycle to share one StarRocks transaction label, so they
become visible atomically, and reduces the number of transactions/labels
from one-per-table to one-per-sink-subtask.
Currently the Flink CDC StarRocks pipeline connector (after FLINK-41006,
connector 1.2.16) does not expose this capability:
* Each table still commits with its own independent transaction label.
* Tables written in the same checkpoint/flush cycle are not visible
atomically, which shows up as an inconsistency window on failure
recovery (converged later by primary-key idempotent replays).
* Wide-table-count sync jobs create many small transactions per flush
cycle, adding FE pressure.
h2. Proposal
Expose the capability behind an opt-in option:
* Add {{sink.transaction.multi-table.enabled}} (default {{false}}) and
{{sink.transaction.multi-table.buffer-size}} options, passed through
to the StarRocks connector. With the option disabled, behavior stays
unchanged.
* Fail fast at pipeline submission when the option is enabled but the
StarRocks server version is below 4.0 or unreachable
({{SELECT CURRENT_VERSION()}} via JDBC).
* Fail fast when schema change events target more than one database,
since a shared StarRocks transaction is single-database scoped.
* When enabled, mark every serialized row with a fixed source partition
and a transaction-end flag, so all tables written by the same sink
subtask within one flush cycle share one transaction label while
keeping the current at-least-once flush cadence.
h2. Limitations (to be documented)
* Requires StarRocks server >= 4.0.
* All target tables must be routed to the same StarRocks database.
* Semantics stays at-least-once; duplicates on failure are deduplicated
by the primary-key table.
* Transaction scope = sink subtask: cross-subtask visibility is not
synchronized (same granularity as the upstream connector).
h2. Depends on
FLINK-41006 (Upgrade StarRocks Flink connector to 1.2.16)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
