sd4324530 opened a new pull request, #4579: URL: https://github.com/apache/flink-cdc/pull/4579
## What is the purpose of this pull request? Exposes the multi-table transaction stream load capability introduced by the StarRocks Flink connector 1.2.16 (upstream PR [#487](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/487), hardened in 1.2.16) behind an opt-in option for the StarRocks pipeline connector. When enabled, all tables written by the same sink subtask within the same flush cycle share one StarRocks transaction label, so they become visible atomically, and the number of StarRocks transactions/labels per flush cycle drops from one-per-table to one-per-subtask, which reduces FE pressure for wide-table-count sync jobs. Depends on #4578 (StarRocks connector upgrade to 1.2.16), please merge that first. ## Brief change log - Add `sink.transaction.multi-table.enabled` (default `false`) and `sink.transaction.multi-table.buffer-size` options, passed through to the StarRocks connector. With the option disabled, behavior is unchanged. - Fail fast at pipeline submission when the option is enabled but the StarRocks server version is below 4.0 or unreachable (JDBC `SELECT CURRENT_VERSION()` check) - Fail fast when schema change events target more than one database, since a shared StarRocks transaction is single-database scoped - When enabled, mark every serialized row with a fixed source partition and a transaction-end flag, so tables written by the same sink subtask within one flush cycle share one transaction label while keeping the current at-least-once flush cadence - Parameterize the test StarRocks container version; add `StarRocksMultiTableTransactionITCase` against StarRocks 4.0.16 and a version-check fail-fast case on the existing 3.5.10 container - Update EN/ZH connector docs with the new options and limitations ## Verifying this change This change added tests and can be verified as follows: - Added unit tests in `StarRocksDataSinkFactoryTest` (version parsing, fail-fast on unreachable server), `EventRecordSerializationSchemaTest` (multi-table row marks, default-marks regression) and `StarRocksMetadataApplierTest` (cross-database rejection) - Added integration tests: - `StarRocksMultiTableTransactionITCase` — end-to-end multi-table atomic write against a StarRocks 4.0.16 container - `StarRocksPipelineITCase#testMultiTableTransactionFailsFastOnUnsupportedServerVersion` — version check fails fast on StarRocks 3.5.10 - Verified all existing unit tests (64) and IT cases (`StarRocksPipelineITCase`, `StarRocksMetadataApplierITCase`) still pass on StarRocks 3.5.10 with the option disabled ## Documentation - Does this pull request introduce a new feature? (yes / no): **yes** - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented): **docs** --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: CodeBuddy (GLM-5.3) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
