cetra3 opened a new pull request, #26282:
URL: https://github.com/apache/datafusion/pull/26282
## Which issue does this PR close?
- Part of #24119.
- Related: #21306, #21308, #20871, #25013, #20589.
This draft shows the whole design working end to end, with benchmarks, so it
can be discussed as a whole before being split into smaller PRs.
## Rationale for this change
Queries that read one field of a struct column read too much from Parquet,
and can't skip row groups or pages using that field's statistics:
- A UDF that reads part of a struct, for example a Variant accessor, decodes
every leaf of the struct, because the reader only recognises `get_field`.
- A filter on a struct field, `WHERE s['a'] > 5`, never skips row groups or
pages. `PruningPredicate` only understands columns, even though Parquet stores
statistics for every nested leaf.
#25013 addresses the first point with two hooks on `ScalarUDFImpl`. This PR
builds on that work, by @peterxcli, and takes a different shape:
- One capability, reached through an accessor method on `ScalarUDFImpl` that
returns `None` by default. Functions without the capability cost one virtual
call; no argument fields are computed for them.
- `get_field` implements the same capability, so the Parquet reader no
longer special-cases `GetFieldFunc`.
- A second method lets a function derive its output statistics from the
statistics of the fields it reads. That gives row group and page pruning for
`get_field` and for other functions, including Variant-style "typed value
unless it fell back" accessors, without Variant knowledge in DataFusion.
## What changes are included in this PR?
**API (`datafusion-expr`)**
```rust
pub type FieldPath = Vec<String>;
pub trait NestedInputAccess {
/// For each argument, the struct field paths this call reads, or `None`
to read it whole.
fn required_fields(&self, args: &ReturnFieldArgs) ->
Vec<Option<Vec<FieldPath>>>;
/// Statistics of the output within one container (row group, page),
derived from
/// the statistics of the paths returned by `required_fields`.
fn output_statistics(&self, args: &ReturnFieldArgs, inputs: &Statistics)
-> Option<ColumnStatistics> { None }
}
// on ScalarUDFImpl (forwarded by ScalarUDF and aliases)
fn as_nested_input_access(&self) -> Option<&dyn NestedInputAccess> { None }
```
`args` describes the call against the schema being planned for: the table
schema at planning, and each file's physical schema when the file is opened.
Answers can differ per file, for example when files shred different Variant
keys.
**`get_field`**
- `required_fields`: the literal key path, when every key names a struct
field. Map lookups and non-literal keys read the whole argument, as today.
- `output_statistics`: the leaf's statistics. Parquet counts a leaf as null
where any ancestor is null, which matches `get_field`.
**Parquet read planning (`projection_read_plan.rs`)**
The `PushdownChecker` branch that downcast to `GetFieldFunc` now handles any
function with the capability. It covers the planning-time filter pushdown
check, the per-file projection mask and the per-file row filter. Behaviour for
`get_field` is unchanged:
- Paths must end at a primitive or an allowed list. Other paths read the
argument whole; subtree selection is a possible follow-up.
- Columns are resolved by name.
- Adapter-inserted struct casts are handled as before.
**Row group and page pruning (`nested_pruning.rs`, `row_group_filter.rs`,
`page_filter.rs`, opener)**
For each file, calls with the capability in the predicate are replaced by
placeholder columns before building `PruningPredicate`, so `PruningPredicate`
and its public interfaces are unchanged.
- When asked for a placeholder's statistics, `RowGroupPruningStatistics`
reads each input leaf's row group statistics with
`StatisticsConverter::from_column_index` and asks the function for its output
statistics. The same masking of untrusted min/max applies as for top-level
columns.
- Page pruning does the same per page. It only combines inputs whose pages
cover the same rows: all input leaves must have identical page boundaries in
the row group.
- The page index is loaded when a placeholder's leaves have one.
**Benchmark**
`benchmarks/src/bin/nested_access.rs` generates a Variant-shaped column and
an equivalent plain struct. It runs queries with the capability off, with
`required_fields` only (how `get_field` behaves on `main`) and with both
methods. It checks that all modes return the same results.
## Benchmarks
1M rows, 50 shredded keys (968 MB file, uncompressed), 8 row groups of
131,072 rows. `k0`/`a0` increase with the row number. Every fourth row group
has one value of `k0` that fell back to the untyped `value`, so statistics
can't be used to skip it. Warm cache, median of 5 runs,
`pushdown_filters=false`, Apple Silicon laptop.
| query | mode | median ms | bytes scanned | row groups pruned | page rows
pruned |
|---|---|---:|---:|---:|---:|
| variant: projection of one key | off | 46.7 | 478,661,700 | 0 | 0 |
| | reads | 2.5 | 9,561,071 | 0 | 0 |
| | reads+stats | 2.4 | 9,561,071 | 0 | 0 |
| variant: `k0 BETWEEN mid AND mid+1000` | off | 46.5 | 478,661,700 | 0 | 0 |
| | reads | 3.9 | 19,611,552 | 0 | 0 |
| | reads+stats | 2.5 | 6,375,966 | 5 | 331,424 |
| variant: top 1% of `k0` | off | 47.3 | 478,661,700 | 0 | 0 |
| | reads | 2.8 | 10,050,481 | 0 | 0 |
| | reads+stats | 2.0 | 2,894,181 | 5 | 280,352 |
| struct: `a0 BETWEEN mid AND mid+1000` | off | 35.8 | 478,579,184 | 0 | 0 |
| | reads (`get_field` on `main`) | 3.3 | 19,608,344 | 0 | 0 |
| | reads+stats | 2.3 | 2,115,986 | 7 | 111,072 |
| `get_field`: `s['a0'] BETWEEN mid AND mid+1000` | this PR | 2.3 |
2,115,986 | 7 | 111,072 |
The Variant filter prunes 5 of 8 row groups rather than 7: the two row
groups holding a fallback value for `k0` are correctly kept.
`pushdown_filters=true` gives similar results. To reproduce:
```sh
cargo run --release --bin nested_access -- --rows 1000000 --keys 50
--pushdown-filters
```
## What is the testing strategy for this PR?
- `projection_read_plan.rs` unit tests: a non-`get_field` function reads
only its declared leaves, the capability is found through aliases, and invalid
or missing paths fall back to reading the whole argument. Filter pushdown
accepts such calls.
- `getfield.rs`: `required_fields` for struct paths, missing fields, paths
below a primitive, non-literal and empty keys, and maps.
- `core/tests/parquet/nested_input_access.rs`, end to end against Parquet
files. Each test compares results with the capability or pruning on and off,
with and without filter pushdown:
- A struct UDF reads far fewer bytes, with identical results for
projection, filter and aggregate queries.
- `get_field` filters prune row groups and pages by the expected counts.
- A Variant-like accessor prunes only row groups and pages where every
value is in the typed leaf. A container holding a fallback value outside the
typed range is read, and its row is returned.
- Existing Parquet unit tests (286), `parquet_integration` (250) and the
related sqllogictest files (projection pushdown, struct, parquet, schema
evolution, filter pushdown, explain) pass unchanged.
## Are there any user-facing changes?
- New public API: `NestedInputAccess`, `FieldPath` and
`ScalarUDFImpl::as_nested_input_access`, all additive with defaults.
- Queries filtering on struct fields can now skip row groups and pages.
## Known gaps and follow-ups
- **Selecting struct subtrees** (paths ending at a struct) currently reads
the argument whole, as for `get_field` today.
- **Nested accessor chains** such as `f(get_field(s, 'x'), ...)` read `s.x`
whole.
## Suggested split
1. `NestedInputAccess::required_fields`, the accessor and `get_field`, used
by the read planner. This replaces #25013's hooks.
2. `output_statistics` plus row group pruning.
3. Page pruning.
4. The benchmark.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
