ranflarion opened a new issue, #26667:
URL: https://github.com/apache/datafusion/issues/26667
### Describe the bug
#24315 taught nested schema pruning's `clip_type` to descend through
`ListView`, `LargeListView`, `FixedSizeList` and `Dictionary`, but scans never
use those clips. `build_read_plan_with_cast_clipping` keeps only the leaf
offsets `clip_for_cast` returns and derives the emitted type with
`type_for_leaf_subset`, whose `project_type_for_leaves` handles only `Struct`,
`List` and `LargeList` and returns `Err(())` for a partial selection under any
other wrapper. The root then falls back to a full read, so a table that
declares a narrower element struct under any of those four wrappers still
decodes every leaf. #24130 added `type_for_leaf_subset` three days before
#24315 merged, and #24315's tests check `clip_for_cast` and the Parquet reader
rather than the read plan.
`type_for_leaf_subset_rejects_unsupported_partial_wrapper` pins the rejection
for `FixedSizeList`.
### To Reproduce
```sql
COPY (
SELECT id, CAST(f AS STRUCT<x INT, pad VARCHAR>[2]) AS f
FROM (VALUES
(1, [named_struct('x', 10, 'pad', 'p1'), named_struct('x', 11, 'pad',
'p2')]),
(2, [named_struct('x', 20, 'pad', 'p3'), named_struct('x', 21, 'pad',
'p4')]),
(3, NULL)
) AS t(id, f)
) TO '/tmp/fixed_size.parquet' STORED AS PARQUET;
CREATE EXTERNAL TABLE narrow (id INT, f STRUCT<x INT>[2])
STORED AS PARQUET LOCATION '/tmp/fixed_size.parquet';
CREATE EXTERNAL TABLE full_schema (id INT, f STRUCT<x INT, pad VARCHAR>[2])
STORED AS PARQUET LOCATION '/tmp/fixed_size.parquet';
EXPLAIN ANALYZE SELECT f FROM narrow;
EXPLAIN ANALYZE SELECT f FROM full_schema;
```
On main (ebf485818) both scans report `bytes_scanned=174.0 B`. A `List`
column narrowed the same way does read less:
`parquet_nested_schema_pruning.slt` pins 172 B against 312 B for its `events`
column.
### Expected behavior
`narrow` reads only `f.x`, as it would through a `List`. With
`project_type_for_leaves` descending the same wrappers as `clip_type`, it reads
83 B.
### Additional context
Part of #24119.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
