dingsongjie opened a new pull request, #12308:
URL: https://github.com/apache/seatunnel/pull/12308
The TiDB-CDC source currently supports syncing only a single table (the
`database-name` + `table-name` pair). This PR adds multi-table sync in one
source via a new `table-names` option, consistent with the option design of the
other cdc-base connectors (MySQL/Postgres/Oracle/SqlServer/Db2:
`exclusive(TABLE_NAMES, TABLE_PATTERN)`).
Changes:
- Add `table-names` option accepting full table names (`db.table`); the
legacy `database-name`/`table-name` pair still works and becomes optional
- Enumerate one split per configured table; round-robin assignment of table
splits to readers
- Isolate per-table transaction buffers
(`preWrites`/`commits`/`committedEvents`): previously they were shared instance
fields, so unresolved transactions of one table could be flushed into another
table's stream (cross-table contamination)
- Fix `ConcurrentModificationException` in `pollNext` by iterating over a
copy of `sourceSplits`, since `addSplits` can structurally modify the list
during restore
- On restore, the subscription follows the latest config: splits and
positions of tables removed from the config are discarded with a WARN; re-added
tables run a fresh snapshot (`enumeratedTables` is retained against the
configured tables)
- Add `TiDBSourceOptions.tableFullName` as the single source of the
`db.table` format; add `hasConfiguredTables` guard for legacy empty configs
- Rename misplaced `SqlServerIncrementalSourceFactoryTest` to
`TiDBSourceFactoryTest` (it already tested `TiDBSourceFactory`)
### Does this PR introduce _any_ user-facing change?
Yes.
- New option `table-names` (a list of `db.table` full names). Example:
```
table-names = ["tidb_cdc.table_2", "tidb_cdc.table_3"]
```
- `database-name`/`table-name` become optional: either the legacy pair or
`table-names` must be present (validated in `createSource`). When only the
legacy pair is used, behavior is unchanged.
- Restore semantics when the config changes between savepoint and restore:
tables removed from the config lose their positions (a WARN is logged) and are
no longer subscribed; a removed-then-re-added table runs a fresh snapshot.
- Fully backward compatible: no option renamed or removed, no default value
changed, so `incompatible-changes.md` does not need an update.
- Connector docs updated in `docs/en` and `docs/zh` (option table,
multi-table example, restore add/remove table semantics).
### How was this patch tested?
- Unit tests (connector-cdc-tidb): 27/27 green, including 3 new tests:
- `transactionBuffersShouldBeIsolatedPerTable` (regression for cross-table
contamination)
- `addSplitsShouldDropSplitsOfTablesRemovedFromConfig`
-
`restoreShouldDiscardStateOfTablesRemovedFromConfigAndReAddRunsFreshSnapshot`
- E2E (connector-cdc-tidb-e2e, Testcontainers): new
`testTiDBCdcMultiTableCheckDataE2E` (snapshot + streaming upsert/delete
convergence on two tables, source-vs-sink exact equality) and
`testTiDBCdcSavepointRestoreWithRemovedTable` (multi-table → savepoint →
restore with one table removed → no stale data for the removed table → re-add →
fresh snapshot). Full `TiDBCDCIT` class: 15/15 green across Flink 1.18 / Flink
1.20 / Zeta; existing single-table tests (incl. savepoint-restore) keep
passing. Await windows widened to 180s for slow Flink deployments in CI.
- `./mvnw spotless:check` clean on both modules.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according to [New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
— N/A, no new dependency
* [x] If necessary, please update the documentation to describe the new
feature.
* [ ] If necessary, please update `incompatible-changes.md` — N/A, fully
backward compatible
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. plugin-mapping.properties — N/A, existing connector
2. seatunnel-dist pom — N/A, existing connector
3. CI label — N/A, existing connector
4. [x] E2E test cases added in seatunnel-e2e
5. plugin_config — N/A, existing connector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
