adriangb opened a new issue, #24119:
URL: https://github.com/apache/datafusion/issues/24119
This EPIC tracks work on making queries over **struct columns in Parquet**
read and evaluate only what they need — projection pruning, filter pushdown,
row-group pruning, and the statistics that feed them.
Scope is the concrete Parquet-side machinery:
`datafusion/datasource-parquet/src/{projection_read_plan,row_filter,nested_schema_pruning}.rs`
and the physical expression adapter's interaction with it. Variant support
builds on some of the same primitives but is tracked separately in #21308.
## Motivation
Two shapes reach the Parquet scan for struct access, and they are handled by
unrelated code:
1. **`get_field` expressions** — `WHERE s['x'] = 200`, `SELECT s['a']`.
Handled by `PushdownChecker` / `StructFieldAccess`.
2. **A narrowed declared schema** — the table declares `s STRUCT<a>` where
the file has `STRUCT<a, b, c>`. The physical expression adapter rewrites the
projected column into `CAST(s AS STRUCT<a>)`, and engines like Spark/Comet
communicate nested projection pruning this way, as a clipped read *schema*
rather than as expressions.
Until #24090, shape (2) read every physical leaf and discarded the extras in
memory. apache/datafusion-comet#4859 reported a production query reading **1.35
TB where plain Spark read 30.9 GB** for the same pruned `ReadSchema`. Any
embedder that hands DataFusion a pre-pruned schema — Comet, delta-rs, Iceberg
integrations — hits the same gap.
#24090 fixes that for the projection path. The items below are what it does
not cover.
## Correctness
- [ ] #24109 — Parquet filter pushdown silently drops a `get_field`
predicate when the file needs schema adaptation (**wrong results**). Blocks the
filter-pushdown work below: on exactly the tables nested pruning targets,
`WHERE s['field'] = ...` with `pushdown_filters = true` is wrong today.
## Projection & filter pruning
- [ ] #24090 — prune unread Parquet leaves when a nested column is cast to a
narrower type (projection path)
- [ ] Extend nested schema pruning to filter pushdown *(this EPIC's first
follow-up; issue linked below)*
- [ ] Union the required leaves when a root is reached by multiple casts, or
by a cast and a `get_field`, instead of falling back to a full read *(linked
below)*
- [ ] Clip `ListView` / `LargeListView` / `Dictionary` / `FixedSizeList` /
`RunEndEncoded` wrappers, which are conservatively left unclipped today
*(linked below)*
- [ ] #23156 — refactor: build a reusable struct-access path tree for
row-filter planning, so projection-mask selection and projected-schema pruning
cannot drift
- [ ] #2581 — introduce `ProjectionMask` to allow nested projection pushdown
(long-standing parent issue for this area)
## Row-group pruning & statistics
- [ ] #20871 — support row group pruning for struct field predicates
- [ ] #8334 — support Parquet statistics for struct columns
- [ ] #20707 — nested types and extension types are not supported in
`Statistics` / `ColumnStatistics`
## Schema evolution
- [ ] #20835 — support additive schema evolution for `List<Struct>` / nested
container types in Parquet scans
## Shared with Variant
- #21306 — all struct-aware optimizations are hardcoded to `GetFieldFunc`; a
custom UDF like `variant_get` gets none of them. Relevant to both this EPIC and
#21308.
## Related
- apache/datafusion-comet#4859 — the production report that motivated the
projection-side work
- #21308 — first-class support for Variant access in Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
