adriangb opened a new pull request, #23392: URL: https://github.com/apache/datafusion/pull/23392
## Which issue does this PR close? - Related to https://github.com/apache/datafusion-comet/issues/4859 (no DataFusion issue filed yet; happy to open one if preferred). See analysis in https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4916909423 and https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4917100051. ## Rationale for this change When a table's logical file schema declares a nested column *narrower* than the physical parquet type — e.g. the table declares `events: LIST<STRUCT<x, y>>` over a file containing `events: LIST<STRUCT<x, y, +16 more fields>>` — DataFusion reads **every leaf** of the column and drops the extra subfields in memory: 1. `DefaultPhysicalExprAdapter` rewrites the projected column into `CAST(events AS narrow_type)`. 2. The projection-mask derivation only understands literal `get_field` chains; the cast's inner column falls into the `ProjectionMask::roots` path, expanding to all physical leaves. 3. The whole column is fetched and decoded; the runtime nested cast (`nested_struct::cast_column`) discards the unrequested subfields. Engines like Spark communicate nested projection pruning to the scan exactly this way — as a clipped read *schema*, not as `get_field` expressions (Spark's `SchemaPruning` rule consumes the field-access expressions and bakes the result into the scan's `requiredSchema`). Comet measured a production query reading **1.35 TB where plain Spark read 30.9 GB** for the same pruned `ReadSchema`. The same behavior is reproducible in pure DataFusion by declaring a narrower nested type in `CREATE EXTERNAL TABLE` and watching `bytes_scanned`. Any embedder that hands DataFusion a pre-pruned schema (Comet, delta-rs, Iceberg integrations) benefits. This PR implements the equivalent of Spark's `ParquetReadSupport.clipParquetSchema`: when a projected root column is consumed only through such a cast, the leaf `ProjectionMask` is computed by walking the physical and cast-target type trees, matching struct fields by name, so only the leaves the cast retains are fetched and decoded. The adapter-inserted cast then degrades to a cheap reorder/rename/null-fill/promotion. **Why key off the cast target instead of the logical-vs-physical schema diff?** The cast target is exactly the contract of the runtime nested cast: `cast_column` consumes source struct children only by looking up *target* field names, recursively through list wrappers, so physical subtrees the target never names are provably dead. It also works for custom `PhysicalExprAdapter`s (which are allowed to inject references to physical-only columns — schema-driven clipping could under-read there, expression-driven clipping degrades safely to a full read), and it requires no opener signature changes. A bare `Column` whose logical type is narrower than the physical file is not a working configuration today, so threading the logical schema would add no coverage. Benchmark results (new `parquet_nested_schema_pruning` bench; wide `list<struct>` file, narrow declared schema): | | bytes_scanned | wall time | |---|---:|---:| | narrow schema, pruning **on** | 3.32 KB | 164 µs | | narrow schema, pruning off (= before this PR) | 25.19 MB | 2.94 ms | | physically narrow file (floor) | 3.32 KB | 150 µs | The `SUM(s['x'])` shape on a schema-evolved struct — where `get_field(CAST(col))` currently defeats the existing expression-level leaf clipping entirely (the bottom-up adapter rewrite means `PushdownChecker` no longer sees a plain `Column` under `get_field`) — improves ~80–90% as well, since it now clips to the cast target. ## What changes are included in this PR? Stacked as four commits, each building green: 1. **bench**: `datafusion/core/benches/parquet_nested_schema_pruning.rs` — registers the same wide file with a narrow and a full declared schema plus a physically-narrow floor, and prints `bytes_scanned` at setup. On `main` the narrow declared schema reads exactly as many bytes as the full one. 2. **refactor (move-only)**: extracts `ParquetReadPlan`, `StructFieldAccess`, `build_projection_read_plan` and the leaf-index/schema-pruning helpers out of the 2100-line `row_filter.rs` into a new `projection_read_plan.rs`. 3. **feat**: the actual change: - New `datafusion/datasource-parquet/src/nested_schema_pruning.rs`: `clip_for_cast` walks physical vs cast-target types (name-matched struct children, case-sensitive to match `cast_column`; recurses through matching `List`/`LargeList` wrappers) and returns kept leaf offsets; `prune_type_by_kept_offsets` derives the Arrow type the reader emits for a kept-offset union. The clip is **total** — maps (no name-based Map arm in `cast_column`), dictionaries, wrapper-kind mismatches, and zero-name-overlap struct levels keep all their leaves, so the worst case is today's full read. Every struct level keeps ≥1 leaf, so struct definition levels (and `s IS NULL` semantics) survive. - `PushdownChecker` optionally collects `CastColumnAccess` (a `CastExpr` over a plain `Column` where `requires_nested_struct_cast(physical, cast_type)`), enabled only for projection analysis — the row-filter path is mechanically untouched. - `build_projection_read_plan` unions cast-clipped leaves with existing `get_field` leaf accesses per root; any whole-column reference wins; roots with only `get_field` accesses keep the bit-identical legacy path. Defensive leaf-count check falls back to a whole-root read if the arrow/parquet leaf mapping ever disagrees. - New config `datafusion.execution.parquet.nested_projection_pruning` (default `true`, kill switch), threaded source → morselizer → opener → `DecoderProjection`, plus proto field + docs + `information_schema` entries. 4. **bench**: pruning-disabled variants so the before/after stays visible in one run. ## Are these changes tested? Yes: - 16 unit tests in the new module, including an arrow-rs roundtrip test that pins the assumption this feature relies on: `ProjectionMask::leaves` over a subset of `List<Struct>` leaves emits exactly the predicted type and preserves null list rows / null struct elements. - 7 integration tests in `datafusion/core/tests/parquet/expr_adapter.rs`, each asserting **identical results** with the flag on and off *and* that `bytes_scanned` drops by more than 2×: list-of-struct narrowing (subset + reorder + `Int32→Int64` leaf promotion + missing-subfield null-fill), top-level struct, struct-level nullability (`s IS NULL`), `get_field` on a narrowed struct, mixed whole-column + subfield access, filter pushdown enabled, and a scan mixing a physically-narrow and a wide file. - New `datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.slt` covering the end-to-end SQL path (`CREATE EXTERNAL TABLE` with a narrower nested type) with the setting toggled both ways. - Full extended test suite passes locally. ## Are there any user-facing changes? A new configuration option, `datafusion.execution.parquet.nested_projection_pruning` (default `true`), documented in `configs.md`. No API changes; behavior is semantically invisible (IO reduction only). Deliberately out of scope, noted for follow-ups: - Map key/value clipping (blocked on a name-based `Map` arm in `nested_struct::cast_column`). - Schema-driven clipping for the row-filter path (filters on evolved structs are not pushed down today, so that is a new capability needing its own analysis, not a regression). - Intersecting `get_field` paths with cast targets (naively unsafe: null-filling a non-nullable logical sibling changes per-batch struct-compatibility validation outcomes). - A pluggable clipping policy (field-id / case-insensitive matching for Iceberg-style embedders), mirroring the expr-adapter factory. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
