SEPURI-SAI-KRISHNA commented on PR #13092: URL: https://github.com/apache/seatunnel/pull/13092#issuecomment-6595043377
@SEZ9 thanks, answering your three points from #13091 here so they sit with the review: 1. One probe per table. `JdbcSourceSplitEnumerator.run` calls `ChunkSplitter.generateSplits` once per table, and that calls `addNullSplitKeySplit` once after the range or hash splits are built. The probe is a single `SELECT 1 ... WHERE <col> IS NULL` with `setMaxRows(1)`, so there is no extra round trip per split. It is skipped when the column is `NOT NULL` or a split already reads the whole table. On a large table with no NULL and no index on the column it can scan the table once, and the EN and ZH docs say so. 2. Shared path. Both JDBC splitters, `FixedChunkSplitter` and `DynamicChunkSplitter`, get `generateSplits` from `ChunkSplitter`. The probe and the NULL split build their SQL from the dialect's `tableIdentifier` and `quoteIdentifier` plus plain `IS NULL`, and limit rows with JDBC `setMaxRows` rather than a `LIMIT` clause. Hash conditions on a nullable column add `AND <col> IS NOT NULL`, so a NULL row can only be read by the NULL split. It runs for real on MySQL (IT) and DuckDB (`FixedChunkSplitterTest`); I have not run it on PostgreSQL, Oracle or SQL Server. 3. Exactly once. `JdbcMysqlSplitIT#testNullSplitKeyRowsAreReadOnce` runs every split and asserts there is exactly one NULL split, that no id is read twice (it fails on the first duplicate), and that all 100 ids are read, 10 of them with a NULL key. It does this for range splits on the automatically chosen unique key and for hash splits on a nullable `partition_column`. `FixedChunkSplitterTest` checks the same on DuckDB, plus no NULL split for a `NOT NULL` column or a nullable column without NULL, and that the NULL split follows `where_condition`. CI on the current head 654d87508c: `JdbcMysqlSplitIT` 6/0 and `FixedChunkSplitterTest` 11/0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
