1fanwang opened a new pull request, #25389: URL: https://github.com/apache/datafusion/pull/25389
## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/25293. ## Rationale for this change Unnesting a NULL struct can expose its stored child values. A non-nullable child field can also make a valid query fail. Both cases now produce NULL values. ## What changes are included in this PR? Output fields include the parent's nullability. The executor applies the parent mask through Arrow and reuses the original child arrays when the parent has no nulls. ## What is the testing strategy for this PR? SQL and native-array regressions cover schema nullability and child validity, including nested and encoded children. Run these commands from the repository root: ```sh cargo run --locked --profile ci -p datafusion-cli -- --quiet --format json \ -c "CREATE TABLE base AS SELECT named_struct('a', 1, 'b', 'x') AS s; CREATE TABLE masked AS SELECT nullif(s, s) AS s FROM base; SELECT unnest(s) FROM masked;" cargo run --locked --profile ci -p datafusion-cli -- --quiet --format json \ -c "SELECT unnest(arrow_cast(NULL, 'Struct(\"a\": non-null Int32)'));" ``` The JSON writer omits NULL-valued fields. Before, the first query exposes values and the second fails: ```text [{"masked.s.a":1,"masked.s.b":"x"}] Error: Arrow error: Invalid argument error: Column '__unnest_placeholder(arrow_cast(NULL,Utf8("Struct("a": non-null Int32)"))).a' is declared as non-nullable but contains null values ``` After the same two commands: ```text [{}] [{}] ``` <details><summary>Raw schema diagnostic</summary> The SQL regression also catches the incorrect output nullability before the fix: ```sh cargo test --locked --profile ci --test sqllogictests -- unnest_struct_nulls.slt ``` Raw before-fix excerpt: ```text 3. query result mismatch: [SQL] SELECT is_nullable FROM information_schema.columns WHERE table_name = 'nullable_fields'; [Diff] (-expected|+actual) - YES + NO ``` </details> ## Are there any user-facing changes? Yes. NULL structs now unnest into NULL fields rather than values or an error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
