li3zhi4 opened a new pull request, #11746:
URL: https://github.com/apache/seatunnel/pull/11746
## Why
The JDBC source splits a table into chunks using **only the first column of
the primary key** (`ChunkSplitter.findSplitKey()` returns the first supported
PK column). For tables with a **composite primary key**, this:
1. **Loses parallelism entirely when the first column has (almost) one
distinct value**: the chunk loop hits `min.equals(max)` and returns a single
full-table chunk — a large table is read by one reader.
2. **Collapses to severely skewed chunks when the first column has low
cardinality / many duplicates**: `nextChunkEnd()` keeps calling `queryMin` to
skip duplicated boundary values; chunk count collapses and one chunk dominates
the job duration.
3. `DynamicChunkSplitter` on `dev` has zero multi-column (`Composite`)
methods — the entire splitter is single-column.
## Scope
- **`connector-jdbc` only** — this is a separate module with its own
splitter (`ChunkSplitter` / `DynamicChunkSplitter`), independent from the CDC
splitter.
- The **CDC-side** composite-key support (base multi-column API + MySQL
first implementation) is a **separate PR** — see the `[Connector-CDC]` issue;
do not bundle both modules here.
- Single-column behavior for single-column keys is unchanged (backward
compatible).
## What changed
### `ChunkSplitter.java`
- `findSplitKey()`: return all supported primary-key columns as the split
key (multi-column `SeaTunnelRowType`) when the PK is composite; user-set
`partition_column` still wins. Single-column PK keeps the exact current result.
### `DynamicChunkSplitter.java`
- Add multi-column `queryMinMaxComposite` / `queryNextChunkMaxComposite` /
`queryMinComposite` and `createCompositeSplit`:
- tuple comparisons **expanded into OR/AND form** (`col1 > ? OR (col1 = ?
AND col2 >= ?)`) so the composite index can be used (a row-constructor `(a,b)
>= (?,?)` frequently cannot);
- `ORDER BY ... LIMIT chunkSize` boundary walking (each call scans at most
`chunkSize` rows);
- half-open `(start, end]` tuple boundary semantics per chunk.
- Wire the composite path into the split generation when the split key is
multi-column (`createDynamicSplits` branches on `splitKey.getTotalFields() >
1`).
- `createDynamicSplitQuerySQL` / `prepareDynamicSplitStatement`: composite
branches (`buildCompositeCondition` / `prepareCompositeStatement`) vs. the
unchanged single-column paths.
## File changes
| File | Change |
|---|---|
| `connector-jdbc/.../source/ChunkSplitter.java` | multi-column
`findSplitKey` (composite PK, dynamic splitter only) |
| `connector-jdbc/.../source/DynamicChunkSplitter.java` |
`queryMinMaxComposite` / `queryNextChunkMaxComposite` / `queryMinComposite` /
`createCompositeSplit` / `buildCompositeCondition` /
`prepareCompositeStatement` + wiring |
| `connector-jdbc/src/test/.../source/CompositeKeyChunkSplitterTest.java` |
new unit tests (composite `findSplitKey`, single-column regression, tuple SQL
shape first/middle/last, user-query wrapping, config parsing) |
| `seatunnel-e2e/.../connector-jdbc-e2e-part-7/.../JdbcMysqlSplitIT.java` |
new `testCompositeKeySplit` e2e (composite-PK table, first column repeats,
asserts multi-chunk + `Object[]` tuple boundaries) |
## Verification
- **Unit tests** (all pass): `CompositeKeyChunkSplitterTest` 5/5,
`DynamicChunkSplitterTest` 5/5 (regression).
- **`mvn spotless:check`**: passes on both modules.
- **Official E2E (Zeta + Testcontainers, MySQL 8.0.43)**:
`JdbcMysqlSplitIT#testCompositeKeySplit` passes — composite-PK table (order_id,
line_no), first column repeats 0/1/2, splits into multiple chunks with
`Object[]` tuple boundaries; single-column numeric/string/date split paths in
the same IT still pass (regression).
## Checklist
- [x] Ported onto `dev` + unit tests pass
- [x] `mvn spotless:check` passes
- [x] Official E2E passes (Zeta + Testcontainers)
- [x] Issue(s) referenced: #11741
- [ ] Docs updated if needed (JDBC connector doc: split-key behavior for
composite PK)
## Note for upstream rebase
`connector-jdbc` splitter on `dev` is single-column and self-contained
(`ChunkSplitter` abstract class + `DynamicChunkSplitter`); no cross-module
dependency on the CDC changes, so this PR can be reviewed/merged independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
