goutamadwant opened a new pull request, #24130: URL: https://github.com/apache/datafusion/pull/24130
## Which issue does this PR close? - Closes #24121. ## Rationale for this change When one projected nested column is consumed through multiple narrowing casts, or through both a narrowing cast and `get_field`, the Parquet read plan currently falls back to reading the entire root column. The leaves required by those consumers can be combined, so reading the full root performs unnecessary I/O. ## What changes are included in this PR? - Accumulate and deduplicate the leaf offsets required by every narrowing cast on a root column. - Add leaves required by `get_field` accesses to the same union. - Derive the Arrow type emitted by the reader from the merged leaf set while retaining a full-read fallback for unsupported partial wrapper shapes. - Add coverage for disjoint, overlapping, and repeated casts, cast plus `get_field`, and casts that diverge below `List<Struct>`. ## Are these changes tested? Yes. The following checks pass: - `cargo fmt --all -- --check` - `cargo test -p datafusion-datasource-parquet --lib` - `cargo test --profile=ci --test sqllogictests -- parquet_nested_schema_pruning.slt` - `cargo clippy -p datafusion-datasource-parquet --all-targets --all-features -- -D warnings` The SQL logic test also verifies that the merged projections return the expected values and scan fewer bytes than a full-root read. ## Are there any user-facing changes? No API or SQL behavior changes. Queries with multiple nested accesses to the same Parquet root can read fewer leaf columns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
