SEPURI-SAI-KRISHNA commented on PR #13092: URL: https://github.com/apache/seatunnel/pull/13092#issuecomment-6822677715
@nzw921rx Thanks, agreed: no NULL split. I will remove it, together with `nullSplitKey` and its state handling, and make automatic split-key selection skip nullable columns, as you suggest. One question before I change the explicit `partition_column` path. Many tables declare the column nullable although it never holds NULL (in MySQL that is the default). Of the 9 Jdbc e2e configs on dev that set `partition_column`, 4 split on a column declared nullable, with `enable_concurrent_read` at its default `true`, so they would fail at start with a rule based on the declared nullability: - `jdbc_mysql_source_and_sink_parallel.conf`: `c_decimal_unsigned_30 DECIMAL(30) unsigned DEFAULT NULL` - `jdbc_mysql_source_and_sink_parallel_upper_lower.conf`: `c_bigint_30 BIGINT(40) unsigned DEFAULT NULL` - `jdbc_postgres_source_and_sink_copy_stmt.conf`: `varchar_col VARCHAR(255)` - `jdbc_starrocks_dialect.conf`: `STRING_COL STRING` All four read with `query`, where the nullability comes from the driver's result set metadata, and the default `JdbcDialectTypeMapper` treats `columnNullableUnknown` as nullable (`isNullable != columnNoNulls`), so a column the driver cannot classify would fail too. Which do you prefer? 1. Fail fast when the explicit `partition_column` is declared nullable, and change those configs to a NOT NULL column (or `enable_concurrent_read = false`), with an `incompatible-changes.md` entry. 2. Fail fast only when the column actually holds a NULL in the read scope, checked with one row-limited `SELECT 1 ... WHERE col IS NULL` during split planning. Jobs whose nullable column has no NULL keep working, a job that would lose rows fails with the same message, and there is still no NULL split or state change. Its cost: on a large table without NULL values and without an index on the column, the check reads the whole table once while the splits are planned. In both, the error names the table and the column and suggests a NOT NULL column or `enable_concurrent_read = false`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
