adriangb opened a new pull request, #23398: URL: https://github.com/apache/datafusion/pull/23398
> [!NOTE] > **Stacked on #23396 (move-only refactor) and #23397 (benchmark).** Until those merge, this PR shows their commits too; the feature itself is the last two commits (`feat: prune unread nested parquet leaves...` and `bench: add pruning-disabled variants...`), ~900 hand-written lines of which roughly half are tests. Supersedes #23392. ## Which issue does this PR close? - Related to https://github.com/apache/datafusion-comet/issues/4859 (no DataFusion issue filed yet; happy to open one if preferred). See analysis in https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4916909423 and https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4917100051. ## Rationale for this change When a table's logical file schema declares a nested column *narrower* than the physical parquet type — e.g. the table declares `events: LIST<STRUCT<x, y>>` over a file containing `events: LIST<STRUCT<x, y, +16 more fields>>` — DataFusion reads **every leaf** of the column and drops the extra subfields in memory: 1. `DefaultPhysicalExprAdapter` rewrites the projected column into `CAST(events AS narrow_type)`. 2. The projection-mask derivation only understands literal `get_field` chains; the cast's inner column falls into the `ProjectionMask::roots` path, expanding to all physical leaves. 3. The whole column is fetched and decoded; the runtime nested cast (`nested_struct::cast_column`) discards the unrequested subfields. Engines like Spark communicate nested projection pruning to the scan exactly this way — as a clipped read *schema*, not as `get_field` expressions (Spark's `SchemaPruning` rule consumes the field-access expressions and bakes the result into the scan's `requiredSchema`). Comet measured a production query reading **1.35 TB where plain Spark read 30.9 GB** for the same pruned `ReadSchema`. The same behavior is reproducible in pure DataFusion by declaring a narrower nested type in `CREATE EXTERNAL TABLE` and watching `bytes_scanned`. Any embedder that hands DataFusion a pre-pruned schema (Comet, delta-rs, Iceberg integrations) benefits. This PR implements the equivalent of Spark's `ParquetReadSupport.clipParquetSchema`: when a projected root column is consumed only through such a cast, the leaf `ProjectionMask` is computed by walking the physical and cast-target type trees, matching struct fields by name, so only the leaves the cast retains are fetched and decoded. The adapter-inserted cast then degrades to a cheap reorder/rename/null-fill/promotion. **Why key off the cast target instead of the logical-vs-physical schema diff?** The cast target is exactly the contract of the runtime nested cast: `cast_column` consumes source struct children only by looking up *target* field names, recursively through list wrappers, so physical subtrees the target never names are provably dead. It also works for custom `PhysicalExprAdapter`s (which are allowed to inject references to physical-only columns — schema-driven clipping could under-read there, expression-driven clipping degrades safely to a full read), and it requires no opener signature changes. A bare `Column` whose logical type is narrower than the physical file is not a working configuration today, so threading the logical schema would add no coverage. Benchmark results (see #23397; wide `list<struct>` file, narrow declared schema): | | bytes_scanned | wall time | |---|---:|---:| | narrow schema, pruning **on** | 3.32 KB | 164 µs | | narrow schema, pruning off (= before this PR) | 25.19 MB | 2.94 ms | | physically narrow file (floor) | 3.32 KB | 150 µs | The `SUM(s['x'])` shape on a schema-evolved struct — where `get_field(CAST(col))` currently defeats the existing expression-level leaf clipping entirely (the bottom-up adapter rewrite means `PushdownChecker` no longer sees a plain `Column` under `get_field`) — improves ~80–90% as well, since it now clips to the cast target. ## What changes are included in this PR? On top of the stacked #23396/#23397: - New `datafusion/datasource-parquet/src/nested_schema_pruning.rs`: `clip_for_cast` walks physical vs cast-target types (name-matched struct children, case-sensitive to match `cast_column`; recurses through matching `List`/`LargeList` wrappers) and returns kept leaf offsets; `prune_type_by_kept_offsets` derives the Arrow type the reader emits for a kept-offset union. The clip is **total** — maps (no name-based Map arm in `cast_column`), dictionaries, wrapper-kind mismatches, and zero-name-overlap struct levels keep all their leaves, so the worst case is today's full read. Every struct level keeps ≥1 leaf, so struct definition levels (and `s IS NULL` semantics) survive. - `PushdownChecker` optionally collects `CastColumnAccess` (a `CastExpr` over a plain `Column` where `requires_nested_struct_cast(physical, cast_type)`), enabled only for projection analysis — the row-filter path is mechanically untouched. - `build_projection_read_plan` unions cast-clipped leaves with existing `get_field` leaf accesses per root; any whole-column reference wins; roots with only `get_field` accesses keep the bit-identical legacy path. A defensive leaf-count check falls back to a whole-root read if the arrow/parquet leaf mapping ever disagrees. - New config `datafusion.execution.parquet.nested_projection_pruning` (default `true`), threaded source → morselizer → opener → `DecoderProjection`, plus proto field + docs + `information_schema` entries. The flag exists purely as an escape hatch / bisection aid for a change in the IO path — there is no scenario where a correctly-working clip should be disabled — and could be deprecated after a release or two. - Pruning-disabled benchmark variants so the before/after stays visible in one run. ## Are these changes tested? Yes: - 16 unit tests in the new module, including an arrow-rs roundtrip test that pins the assumption this feature relies on: `ProjectionMask::leaves` over a subset of `List<Struct>` leaves emits exactly the predicted type and preserves null list rows / null struct elements. - 7 integration tests in `datafusion/core/tests/parquet/expr_adapter.rs`, each asserting **identical results** with the flag on and off *and* that `bytes_scanned` drops by more than 2×: list-of-struct narrowing (subset + reorder + `Int32→Int64` leaf promotion + missing-subfield null-fill), top-level struct, struct-level nullability (`s IS NULL`), `get_field` on a narrowed struct, mixed whole-column + subfield access, filter pushdown enabled, and a scan mixing a physically-narrow and a wide file. - New `datafusion/sqllogictest/test_files/parquet_nested_schema_pruning.slt` covering the end-to-end SQL path (`CREATE EXTERNAL TABLE` with a narrower nested type) with the setting toggled both ways. - Full extended test suite passes locally. ## Are there any user-facing changes? A new configuration option, `datafusion.execution.parquet.nested_projection_pruning` (default `true`), documented in `configs.md`. No API changes; behavior is semantically invisible (IO reduction only). Deliberately out of scope, noted for follow-ups: - Map key/value clipping (blocked on a name-based `Map` arm in `nested_struct::cast_column`). - Schema-driven clipping for the row-filter path (filters on evolved structs are not pushed down today, so that is a new capability needing its own analysis, not a regression). - Intersecting `get_field` paths with cast targets (naively unsafe: null-filling a non-nullable logical sibling changes per-batch struct-compatibility validation outcomes). - A pluggable clipping policy (field-id / case-insensitive matching for Iceberg-style embedders), mirroring the expr-adapter factory. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
