andygrove opened a new issue, #7037: URL: https://github.com/apache/datafusion-comet/issues/7037
### What is the problem the feature request solves? The native Parquet scan never prunes row groups or pages with a predicate on a struct field, such as `WHERE s.k = 31337`, so a selective nested filter reads the whole file. Spark pushes these predicates into Parquet by default (`spark.sql.optimizer.nestedPredicatePushdownSupportedFileSources` includes `parquet`), and parquet-mr prunes row groups using the leaf's statistics. Measured on `main` at 66c63bd307 (Spark 4.1.3) over one Spark-written file of 40,000 rows in 37 row groups, with `id` and `s.k` both increasing. The numbers are the native scan's metrics: | Filter | `row_groups_pruned_statistics` | `page_index_pages_matched` | `bytes_scanned` | | --- | --- | --- | --- | | `id = 31337` | 36 of 37 | 1 | 4,450 | | `s.k = 31337` | 0 of 37 | 0 | 322,805 | | `s.k IS NULL` | 0 of 37 | 0 | 322,805 | `spark.comet.parquet.rowFilterPushdown.enabled=true` gives the same numbers. DataFusion 55's pruning predicate does not resolve a struct field access to its leaf statistics (apache/datafusion#20871, with apache/datafusion#25013 open). Comet also sends struct field access as its own `GetStructField`, evaluated by ordinal (`native/core/src/execution/planner.rs` lines 1112-1115), rather than DataFusion's `get_field`, so the data filters would also need translating before DataFusion could prune with them. ### Describe the potential solution Once DataFusion supports it, emit struct field predicates in the data filters in a form DataFusion's pruning recognizes, and check the pruning metrics against top-level predicates. ### Additional context Found in a nested-type audit of the native scans. #5553 and #5739 depend on this for their TIMESTAMP_MILLIS overflow parity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
