andygrove opened a new issue, #7036: URL: https://github.com/apache/datafusion-comet/issues/7036
### What is the problem the feature request solves? #4859 taught the native Parquet scan to read only the Parquet leaves Spark's nested schema pruning asks for, but only for some shapes. For the shapes below the scan still reads the whole top-level column, so a query that needs one small field of a wide struct pays for every field. The adapter keeps DataFusion's `CastExpr`, which DataFusion can clip to the needed leaves (`projection_read_plan.rs`, `nested_schema_pruning.rs`), only when `is_pure_structural_narrowing` (`native/core/src/parquet/schema_adapter.rs` lines 111-187) holds. That requires every requested struct field to have exactly one exact-name match, no field ids, no map anywhere in the requested type, and leaf types equal to the file's. Anything else becomes `CometCastColumnExpr`, which DataFusion cannot see into, so it reads the whole column. DataFusion never clips map values either. Measured on `main` at 66c63bd307 (Spark 4.1.3), with 20,000 Spark-written rows per file, where each struct holds a small int field and a large string field. "Narrow" is the query on the left, "whole" is reading the whole column, and the numbers are the native scan's `bytes_scanned`: | Query | Narrow | Whole | Pruned? | | --- | --- | --- | --- | | `s.small` (control) | 80,044 | 4,357,215 | yes | | `arrs.small` over `array<struct>` (control) | 80,046 | 4,343,590 | yes | | `s.t`, a TIMESTAMP_MICROS leaf (control) | 158,616 | 4,505,215 | yes | | `sm.mm`, a map inside a struct | 4,362,042 | 4,362,042 | no | | `ms['k'].small`, a map value struct | 4,345,297 | 4,345,297 | no | | `map_values(ms).small` | 4,345,297 | 4,345,297 | no | | `S.SMALL` with `caseSensitive=false` over a file with `s.small` | 4,357,215 | 4,357,215 | no | | `s.small` with `spark.sql.parquet.fieldId.read.enabled=true` | 4,353,017 | 4,353,017 | no | | `s` read as `struct<small, newfield>` (a requested field missing from the file) | 4,505,215 | 4,505,215 | no | | `s` read as `struct<small: bigint>` (type widening) | 4,505,215 | 4,505,215 | no | | `s.t`, a TIMESTAMP_MILLIS leaf | 4,426,688 | 4,426,688 | no | | `s` read as `struct<t: timestamp_ntz>` over a `timestamp` leaf | 4,505,215 | 4,505,215 | no | Reading the code, the same applies to any other requested leaf whose type is not an exact match: unsigned ints, ENUM read as string, BINARY read as string, decimal256, and files whose `ARROW:schema` hint changes a leaf type (`LargeUtf8`, view types, dictionaries, or a timestamp timezone other than UTC, which pyarrow and other Arrow writers produce). ### Describe the potential solution Options, not exclusive: - Clip the Parquet projection to the requested leaves before casting, so that whatever cast runs afterwards sees only those leaves. This is what Spark does with `clipParquetSchema`, and it would cover every shape above. - Widen the kept-cast path to exact-type maps and to struct fields resolved case-insensitively or by field id, where DataFusion's cast agrees with Spark. - Teach DataFusion to prune leaves under casts and map values. ### Additional context Found in a nested-type audit of the native scans. #4859 is the closed issue for the original shape, and #6724 is the same problem in the native Iceberg scan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
