rgbuilds opened a new pull request, #26195: URL: https://github.com/apache/datafusion/pull/26195
## Which issue does this PR close? - Closes #26172. - Depends on #26104. This draft is based on its head (`cf0cf420f4daca37186e748014d896a28855cc74`); the current diff includes that PR. I will rebase onto main after it merges. ## Rationale for this change When a required child is missing from one Parquet file but its containing struct exists in both, inference leaves the child non-nullable. Selecting the child fails, and an `IS NULL` predicate can incorrectly return no rows. With this change, selecting `s.y` returns `10` and NULL, and `WHERE s.y IS NULL` returns the row from the file without `y`. ## What changes are included in this PR? Extend partial-field nullability correction recursively through structs, lists, and large lists. Count children only across schemas containing their parent. An absent parent widens the parent; it does not by itself widen its required children. Required list elements remain required while missing struct children within those elements become nullable. Preserve existing nullability and metadata. Other container types retain the existing merge behavior. Optimized synthetic benchmarks (`release-nonlto`, Apple M4 Mac Mini, in-memory Parquet files, warmed metadata cache, three process runs per variant) against the exact #26104 head measured approximately 2.4–3.0% additional schema-inference time for evolving nested schemas: 32 files × 128 children took 1.973 → 2.033 ms; 256 files × 128 children took 16.072 → 16.459 ms. Flat-schema aggregate medians showed no consistent inference penalty. Registered-table query median differences ranged from -0.15% to +1.12% (queries use universally present columns, since missing-child queries fail on the baseline); these small synthetic measurements do not establish production query impact. The change adds inference traversal and allocations, with no new footer reads or scan-time correction pass. ## What is the testing strategy for this PR? - Unit tests cover absent parents, independent nested paths, metadata, and required list elements. - A Parquet integration regression checks struct/list/large-list projection and `IS NULL` across 12 configurations (metadata and pushdown on/off). It fails on the unfixed dependency head. - SQL regression verifies the missing child's projected NULL and the matching `IS NULL` row. - Fresh extended workspace validation passed all Rust library/integration/bin tests and all 529 SQL logic files. The 126 extended fuzz tests passed in the initial run; subsequent Rust runs skipped only those already-passing cases. All-target/all-feature Clippy with warnings denied, formatting, and every submission lint check passed. macOS emitted linker unwind-table warnings when linking large test executables. ## Are there any user-facing changes? Inferred nested fields missing from some containing schemas become nullable, allowing missing values to be read and `IS NULL` queries to return correct rows. Explicit schemas and public APIs are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
