zhangshenghang opened a new pull request, #12160:
URL: https://github.com/apache/seatunnel/pull/12160
## Purpose
The configured `where_condition` was applied only to the read SQL built by
`createSingleSplitStatement`. The split layout itself was computed against the
unfiltered table, so an Oracle -> SelectDB job with a 2-hour incremental window
on a 77M-row table was split into 777 chunks covering the full primary-key
range, then read serially. The reader spent most of its time scanning
mostly-empty chunks, even though the read SQL was already correctly filtered.
This change applies `where_condition` to the split metadata queries as well,
so the chunk count becomes driven by the matched rows instead of the full
primary-key range.
## Changes
- `JdbcSourceTable`: add `withQuery(String)` copy method that preserves
every other field, so callers can swap in a wrapped query without losing
`catalogTable`, `partitionColumn`, etc.
- `ChunkSplitter`:
- Add `applyWhereCondition(JdbcSourceTable)`: returns a new table whose
query is wrapped as `SELECT * FROM (<base>) tmp <where_condition>`, or returns
the original table when no where condition is set.
- Call `applyWhereCondition` in `validateStringRangeSplit` (string split
key sampling validation), `queryMin`, and `queryMinMax`.
- Extend the split-end log to report whether a where condition is in
effect for the split metadata queries, and warn when a table produces more than
512 chunks.
- `DynamicChunkSplitter`:
- Call `applyWhereCondition` in `sampleDataFromColumn` (used by the
uneven-distribution path), in `queryApproximateRowCnt`, and in `nextChunkEnd`
(used by `splitUnevenlySizedChunks` for boundary probes).
- On Oracle, the row count estimation already switches to `COUNT(*)` when
the inner query contains a `WHERE` clause, so applying the wrapper removes the
need for a separate approximate-row-count branch for filtered reads.
- `JdbcSourceReader`: track assigned and processed split counts and log a
progress line every 50 processed splits so a long job's movement is visible.
The read SQL is unchanged: `createSingleSplitStatement` still appends the
where condition to the original query, so the executed read SQL is
byte-identical to before. Only the split layout becomes correct.
## Validation
```
./mvnw -pl seatunnel-connectors-v2/connector-jdbc \
-Dtest=DynamicChunkSplitterTest \
-Dcheckstyle.skip -Dspotless.check.skip test
```
Result: `Tests run: 8, Failures: 0, Errors: 0, Skipped: 0`
The new tests cover:
- `applyWhereCondition` is a no-op when no where condition is set (returns
the same table instance).
- The user query is wrapped as `SELECT * FROM (<query>) tmp <where>`.
- When no user query is configured, the table identifier is used as the
wrapped base.
## Impact
- Behavior change: with a configured `where_condition`, the source now
produces a chunk count proportional to the matched rows instead of the full
primary-key range. Jobs without `where_condition` are unchanged. Other dialects
are unaffected because `applyWhereCondition` returns the original table when no
where condition is set.
- No new public API. The new `withQuery` is package-private in spirit; it is
a `public` method on a `public` data class so it can be reached from test code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
