SEPURI-SAI-KRISHNA opened a new issue, #13091: URL: https://github.com/apache/seatunnel/issues/13091
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened The JDBC source silently skips every row whose split column is NULL once a table is read by more than one split. The job finishes without an error or a warning. Each split reads a range or a hash bucket of the split column: `col <= ? AND NOT (col = ?)`, `col >= ? AND NOT (col = ?) AND col <= ?`, `col >= ?` (dynamic splitter), `col >= ? AND col <= ?` (fixed splitter), or `ABS(CRC32(col) % n) = ?` (hash split of a string column). None of these conditions matches NULL, and no split reads `col IS NULL`. The split column can be NULL in two ways: 1. **Without any split configuration.** For a table without a primary key, `ChunkSplitter#findSplitKey` takes the first column of a unique index, without checking whether the column is nullable. MySQL allows any number of NULLs in a unique index (as in the table below), and so does PostgreSQL by default. 2. **With `partition_column`** on a nullable column, with either the fixed or the dynamic splitter. This is the JDBC source counterpart of #12598, which #12597 fixed for the JDBC-based CDC sources only (`connector-cdc-base`). The JDBC source has its own splitter in `connector-jdbc` and is unchanged. ### SeaTunnel Version dev (b3db27900b) and the 3.0.0 release. With the released `connector-jdbc` 3.0.0 from Maven Central, the dynamic splitter reads 90 of 100 rows from a table like the one below (no primary key, 10 rows with a NULL unique key). ### SeaTunnel Config MySQL 8.0.43 (`mysql:8.0.43`, the image of `JdbcMysqlSplitIT`), a table without a primary key and with a nullable unique key: ```sql CREATE TABLE ns.u ( id INT NOT NULL, code INT NULL, sname VARCHAR(16) NULL, UNIQUE KEY uk_code (code) ); -- 1000 rows, id 1..1000; code is NULL for every 10th row (100 rows), sname for every 7th row (142 rows) ``` ```hocon env { parallelism = 2 job.mode = "BATCH" } source { Jdbc { url = "jdbc:mysql://127.0.0.1:13306/ns" driver = "com.mysql.cj.jdbc.Driver" username = "root" password = "..." table_path = "ns.u" split.size = 100 } } sink { LocalFile { path = "/tmp/out" file_format_type = "text" } } ``` ### Running Command Zeta local mode, through `SeaTunnelEngineLocalExample` from `seatunnel-examples/seatunnel-engine-examples` with the config above, on JDK 17. ### Error Exception There is no exception. The log shows the unique key chosen as the split column: ``` Splitting table ns.u into chunks, split column: code, min: 1, max: 999, chunk size: 100 ... Split table ns.u into 11 chunks ``` and the sink receives 900 of the 1000 rows. The missing ids are exactly the 100 rows with `code IS NULL`. The same table read with other split settings, each a real job on the Zeta engine: | Source settings | Rows read | Missing | | --- | --- | --- | | `query`, no `partition_column` (one split) | 1000 | none | | `table_path`, no split settings (above) | 900 | the 100 NULL `code` rows | | `table_path`, `partition_column = "code"` | 900 | the 100 NULL `code` rows | | `query`, `partition_column = "sname"`, `partition_num = 4` (hash split) | 858 | the 142 NULL `sname` rows | | `query`, `partition_column = "sname"`, `split.string_split_mode = "charset_based"` | 858 | the 142 NULL `sname` rows | | `table_path`, `partition_column = "sname"`, `split.string-strategy = "hash"` | 858 | the 142 NULL `sname` rows | The fixed and dynamic numeric splits and the sampled string splits lose the same rows on an embedded DuckDB table. ### Zeta or Flink or Spark Version Zeta (local mode). The split conditions are built by the connector, so the engine does not matter. ### Java or Scala Version JDK 17 ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
