adriangb opened a new issue, #24120:
URL: https://github.com/apache/datafusion/issues/24120
Part of #24119.
## Background
apache/datafusion#24090 teaches the Parquet **projection** path to prune
leaves when a nested column is consumed through a narrowing cast — the shape
`DefaultPhysicalExprAdapter` produces whenever a table's declared nested schema
is narrower than the physical file (Comet, delta-rs, Iceberg integrations all
hit this). `PushdownChecker` collects a `CastColumnAccess` for `CAST(col AS
narrower_nested_type)`, and `clip_for_cast` walks the physical and target type
trees together to compute exactly which Parquet leaves the cast consumes.
That collection is deliberately opt-in and enabled **only for projection
analysis**:
```rust
/// Enable collection of whole-column casts to narrower nested types.
pub(crate) fn with_cast_collection(mut self) -> Self { ... }
```
Filter pushdown leaves it off, so a predicate over a narrowed nested column
still reads every physical leaf of that column.
## What this issue asks for
Extend the same clipping to the row-filter path
(`datafusion/datasource-parquet/src/row_filter.rs`), so that with
`pushdown_filters = true` a predicate like `WHERE s['x'] = 200` against a table
whose declared schema narrows `s` reads only the leaves the predicate needs,
rather than the whole column.
The projection side already computes this; the filter side needs the
equivalent two outputs kept in agreement:
1. the `ProjectionMask` handed to the reader for the filter, and
2. the projected Arrow schema the row filter evaluates against.
## Prerequisite: apache/datafusion#24109
This cannot land as a pure optimization until #24109 is resolved. Today
`PushdownChecker::f_down` only recognises `get_field` whose first argument is a
bare `Column`. When the expression adapter interposes a cast — which is
*always* the case on exactly the tables this feature targets — the visitor
descends past the `CastExpr`, sees a bare struct `Column`, sets
`non_primitive_columns = true`, and `pushdown_columns` returns `None`. The
conjunct is then silently dropped from the row filter while planning has
already removed `FilterExec` on the strength of an "exactly handled" claim made
against the *table* schema.
So filter pushdown over a narrowed nested column is currently **unsound, not
merely unoptimised**. Teaching `PushdownChecker` to look through the cast would
address both #24109 and this issue, but only if the post-decode fallback
proposed in #24109 exists to catch whatever the row-filter construction still
cannot handle.
## Suggested order
1. Fix #24109 (post-decode filter fallback, so a failed row-filter build
costs performance rather than correctness).
2. Enable cast collection in the filter path and route clipped leaves
through row-filter planning.
3. Reconcile with the struct-access path handling in `row_filter.rs` — see
#23156, which proposes a shared access tree for exactly the "mask and projected
schema must describe the same pruned shape" invariant.
## Notes
- Related: apache/datafusion-comet#4859, the production report that
motivated the projection-side work (1.35 TB read vs 30.9 GB for the same pruned
`ReadSchema`).
- The clip is total by design: any shape it does not understand keeps all
leaves, so the worst case is today's full read. That property should be
preserved on the filter side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
