andygrove opened a new issue, #7038: URL: https://github.com/apache/datafusion-comet/issues/7038
### What is the problem the feature request solves? The native Iceberg scan never pushes a residual predicate on a struct field, such as `s.k = 5`, into iceberg-rust, so no row group or page inside a data file is skipped for it. Iceberg's own reader binds nested struct fields in its row-group filters (`ParquetMetricsRowGroupFilter`, `ParquetDictionaryRowGroupFilter`), so it can skip them. File-level pruning from manifest metrics still happens during Iceberg planning on the JVM, for both engines. Two things stop it today: - `CometIcebergNativeScan.icebergExprToProto` matches a residual's column name against the scan's top-level output attributes. Iceberg names a nested reference by its dotted path (`s.k`), which never matches, so the predicate is dropped (comment around lines 706-709). An AND with any such conjunct drops the whole residual. #5883 covers keeping the other conjuncts. - iceberg-rust's Arrow row filter only supports top-level columns: `project_column` returns `Does not support struct column yet.` for a struct (`crates/iceberg/src/arrow/reader/predicate_visitor.rs` around line 266 at the current pin). ### Describe the potential solution Push struct-field residuals once iceberg-rust's row-group, page index and row filter evaluation supports nested references, and keep the post-scan filter exact as today. Predicates on list elements and map values stay out, as in Iceberg. ### Additional context Found in a nested-type audit of the native scans. #6854 is the same kind of gap for decimal columns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
