timmyb32r opened a new pull request, #12256: URL: https://github.com/apache/arrow-rs/pull/12256
# Which issue does this PR close? None — standalone bug fix. # Rationale for this change `Schema::normalize` discards the nullability of expanded parent structs. A non-nullable child of a nullable struct therefore becomes a non-nullable flattened field, even though a null parent makes that child's normalized value null. For example: ```text Before: parent: nullable Struct<child: non-nullable Int32> Previously normalized: parent.child: non-nullable Int32 Expected: parent.child: nullable Int32 ``` This can produce a schema that rejects the corresponding data from `RecordBatch::normalize`. # What changes are included in this PR? Propagate accumulated parent nullability during schema normalization and combine it with each field's own nullability. Fields retained at the depth limit also inherit nullability from expanded ancestors. Nested fields inside retained structs and list element nullability are preserved. # Are these changes tested? Added a regression test covering all combinations of parent, intermediate struct, and child nullability, with scalar and list fields and multiple depth limits. It also checks that nullability does not leak into unrelated sibling fields. Extended the nullable-struct record batch test to verify that `Schema::normalize` produces the expected schema and that this schema accepts the normalized columns. Both regressions fail before the fix and pass afterward. Validation passed: - `cargo test -p arrow-schema -p arrow-array --all-features`. - `cargo test --workspace`. - Targeted schema and record batch regression tests under Miri. - `cargo clippy --workspace --all-targets --all-features -- -D warnings`. - Formatting and spelling checks. # Are there any significant user-facing changes? Flattened fields now correctly become nullable when an expanded ancestor is nullable, including when normalization stops at a depth limit. There are no public API signature changes. # AI assistance Codex helped implement the fix and regression tests, run validation, and draft this PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
