bvolpato opened a new pull request, #24951: URL: https://github.com/apache/datafusion/pull/24951
## Which issue does this PR close? Related to #18083. Full support for NULL partition values remains covered by that issue. ## Rationale for this change Partitioned writes read partition values without checking their validity. A NULL integer partition key can be written under `p=0`, so reading the output turns NULL into zero and silently changes the data. For example: ```sql CREATE TABLE src(p INT, v INT) AS VALUES (NULL, 1), (0, 2); COPY src TO '/tmp/null_partition/' STORED AS PARQUET PARTITIONED BY (p); ``` Both rows can end up in the zero partition. This PR rejects the write with `NULL values are not supported for partition column 'p'`. ## What changes are included in this PR? - Check logical nulls before extracting partition keys, including NULL dictionary values referenced by valid dictionary indices. - Continue accepting nullable columns when the selected rows contain no NULL values. - Add focused unit tests and a `copy.slt` regression covering rejection and a filtered successful write. ## What is the testing strategy for this PR? - Both new rejection tests fail against upstream production code at `35f58f53cde2f634c21f6370e385cf1d7e9bc55c` and pass with the fix. - `cargo test --locked -p datafusion-datasource --lib`: 180 passed. - `cargo fmt --all -- --check` and `git diff --check` passed. - Draft pending all-target/all-feature Clippy, the extended workspace suite, and execution of `copy.slt`. ## Are there any user-facing changes? Partitioned writes containing NULL partition keys now return an execution error. Callers can filter or replace these values before writing. Validation happens per batch, so files written by earlier batches can remain after an error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
