ErikBPF opened a new issue, #5731: URL: https://github.com/apache/datafusion-comet/issues/5731
## Describe the bug `CometScanRule` declines every Iceberg scan whose pushed filters contain an `IS NULL`/`IS NOT NULL` predicate on a complex-typed column, with the reason: ``` IS NULL / IS NOT NULL predicates on complex type columns (struct/array/map) are not yet supported by iceberg-rust ``` The check (added in #2528) predates the current iceberg-rust pin (bumped in #5651). iceberg-rust's Arrow predicate visitor now evaluates `is_null`/`is_not_null` on **list** and **map** columns through arrow's native `is_null`/`is_not_null` (`crates/iceberg/src/arrow/reader/predicate_visitor.rs`); only **struct** columns are rejected (`project_column` errors on `DataType::Struct`). ## Impact The check is a false positive for list columns, which is a common real pattern: Spark's optimizer pushes `isnotnull(arr)` below `Generate` for `explode(arr)`: ``` BatchScan local_tpch.orders_nested [COMET: IS NULL / IS NOT NULL predicates on complex type columns ...] ``` Each declined scan forces the whole plan back to Spark and cascades into further fallbacks (`Comet aggregate that merges intermediate buffers requires a Comet child aggregate ...`). On a 24-query derived TPC-H benchmark (SF1, 3 runs, cold caches, Spark 4.1.3, Iceberg hadoop catalog, table with `array<struct>` line-item column): | metric | check active | list false-positive removed | |---|---:|---:| | Comet/Iceberg total (24q) | 20.63 s | 16.93 s (−17.9%) | | `q1_nested_explode` | 2.20 s | 0.77 s (−65%) | | `e14_nested` | 2.03 s | 0.86 s (−58%) | | Comet/Iceberg vs Comet/Parquet total | +26% | +3.7% | Parity vs vanilla Spark verified on all 24 queries in both configurations. A fix that narrows the check to struct-only columns will be submitted as a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
