SEPURI-SAI-KRISHNA commented on PR #13092: URL: https://github.com/apache/seatunnel/pull/13092#issuecomment-6829899464
@nzw921rx Thanks, that works for me for an explicit `partition_column`: no check, no failure, and a clear warning in the docs. Before I rework the PR, one point to confirm about the column chosen automatically, because a doc warning cannot help there. With `table_path`, no `partition_column` and no primary key, the splitter takes the first supported column of a unique key. A unique key can hold NULL (MySQL, PostgreSQL, Oracle, SQL Server), so such a table loses its NULL rows although the user configured nothing about splitting, and there is nothing in the config to change after reading a warning. Skipping nullable columns there is what you suggested earlier: it needs no query, never fails a job, and the table is split on a NOT NULL unique key column if it has one, otherwise read as one split, like a table without a key today. So the PR would become: 1. Automatic split key selection skips nullable columns. Primary key columns are NOT NULL, so tables with a primary key keep their splits. 2. An explicit `partition_column` keeps today's behavior. 3. `partition_column` in Jdbc.md (EN and ZH) and the option description say what it is for, the supported types (numeric, string, date), when it is used (with a top-level `query` it selects the fixed splitter with `partition_num`; with `table_path` it replaces the automatic key), that it must be in the query result, and the warning: with most dialects, rows whose value is NULL are not read when the table is read by more than one split, so use a NOT NULL column or `enable_concurrent_read = false`. (The CockroachDB dialect already keeps NULL rows in hash splits, #12828, so I would note that exception.) 4. The NULL split, `nullSplitKey`, the state change and the NULL probe are removed. `incompatible-changes.md` only describes item 1: a table that was split on a nullable unique key column is now split on a NOT NULL unique key column, or read as one split if it has none. I would leave out a WARN log for a nullable explicit column, since many columns are declared nullable without holding NULL and it would be noisy, unless you want it. Is this the scope you have in mind? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
