SEPURI-SAI-KRISHNA commented on PR #13092: URL: https://github.com/apache/seatunnel/pull/13092#issuecomment-6568483205
Thanks @DanielLeens for the review. Both points are addressed in the new commit. **Issue 1 (probe cost).** The `Jdbc` source docs (EN and ZH) now say that the check is one `SELECT 1 ... WHERE <column> IS NULL` query per table, run once at split planning, that it stops at the first NULL, and that on a large table without NULL values it reads the whole table unless the database can answer `IS NULL` from an index on the split column. The wording is not tied to one database, so it also holds for the Oracle case you mentioned. A column declared `NOT NULL` skips the check. I kept the probe as its own query instead of folding it into MIN/MAX: on an indexed column MIN and MAX read only the two ends of the index, and adding a NULL check to the same query would make that query scan the table. **Issue 2 (`where_condition`).** New `FixedChunkSplitterTest#testNullSplitKeySplitFollowsWhereCondition`: - `WHERE id > 50`, once with a narrow `query` (`SELECT id, code FROM t`) and once with only the table path: ids 51 to 100 must be read exactly once, NULL rows included, by one NULL split. If either branch of the NULL split statement ignores the condition, it fails with 55 rows instead of 50. - `WHERE name IS NOT NULL`, on a column the query does not select: no matching row has a NULL `code`, so no NULL split may be added. If the probe ignores the condition, it fails because a NULL split is added. I broke each of the three spots on purpose to confirm the test fails. `connector-jdbc` unit tests: 1201, the only errors are the 4 Oracle and Kingbase container tests that need Docker here. Spotless is clean. **CI.** The two failed jobs were `unit-test (11/17, ubuntu-latest)`, both in `seatunnel-engine-server` (`TaskTest` and `PhysicalVertexTest`, "Node failed to start!" after 300 s). This PR does not touch the engine. `JdbcMysqlSplitIT` ran in `updated-modules-integration-test-part-8` (6 tests, 0 failures). The new commit starts a fresh run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
