adriangb opened a new issue, #24122: URL: https://github.com/apache/datafusion/issues/24122
Part of #24119. ## Background apache/datafusion#24090 added `datafusion/datasource-parquet/src/nested_schema_pruning.rs`, which clips a nested column's Parquet leaves to the subset a narrowing cast actually consumes. `clip_type` recurses through `Struct`, `List` and `LargeList`, and keeps every leaf for anything else: ```rust // Anything else, leaf pairs, wrapper-kind mismatches, maps, // dictionaries, fixed-size lists, views, is kept wholesale. _ => keep_all_leaves(physical, next_leaf, kept), ``` The module doc records this as deliberate: > Nor are `ListView`/`LargeListView`/`Dictionary` wrappers clipped here, even though `cast_column` does recurse through them by name. That is a conservative choice (safe, since the worst case is still just a full read) left as a candidate follow-up rather than something this module currently handles. So a column typed `ListView<Struct<a, b, c>>` or `Dictionary<_, Struct<a, b, c>>` still reads all three leaves under `CAST(col AS ListView<Struct<a>>)`, where the equivalent `List<Struct<...>>` reads one. ## What this issue asks for Extend `clip_type` to recurse through the wrapper kinds where the runtime cast provably consumes children by name: - `ListView` / `LargeListView` — `cast_column` recurses through them the same way it does `List` / `LargeList` - `FixedSizeList` - `Dictionary` (clip the value type) - `RunEndEncoded` (clip the value type) `nested_child` in the same module already enumerates exactly this set for `count_leaves` / `contains_struct`, so the leaf accounting side is in place; what's missing is the matching arms in `clip_type` that rebuild the wrapper around the pruned child type. **Explicitly out of scope: `Map`.** Map values are not clippable — the runtime cast routes maps through Arrow's positional struct cast, which requires all children to be present. That must stay a full read. ## Requirements - Preserve the existing invariant that every clipped struct level keeps at least one leaf (the reader reconstructs ancestor validity from the definition levels of surviving leaves), and the `unclippable` bail-out when it cannot be met. - Preserve the total-fallback property: an unrecognised shape keeps all leaves, so the worst case stays today's full read. - Wrapper-kind *mismatches* between physical and target (e.g. physical `List`, target `ListView`) must keep falling through to a full read. ## Test coverage to add Mirroring the existing unit tests in `nested_schema_pruning.rs`: - `ListView<Struct<..>>` and `LargeListView<Struct<..>>` narrowed by a cast - `FixedSizeList<Struct<..>>` - `Dictionary<_, Struct<..>>` and `RunEndEncoded<_, Struct<..>>` - wrapper-kind mismatch → no clipping - `Map` → still no clipping (guard against a regression that starts clipping map values) - an arrow-rs roundtrip test pinning that `ProjectionMask::leaves` over the predicted leaf subset emits exactly the predicted type, as the `List<Struct>` test does today ## Notes These wrapper types rarely appear as Parquet-derived Arrow types in practice, so this is a completeness item rather than a hot path. Filing so the deliberate gap is tracked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
