ErikBPF opened a new pull request, #5732: URL: https://github.com/apache/datafusion-comet/pull/5732
## Which issue does this PR close? Closes #5731 ## Rationale for this change The Iceberg scan-rule check that declines scans carrying `IS NULL`/`IS NOT NULL` predicates on complex-typed columns was written against an older iceberg-rust pin (#2528). At the current pin (#5651), iceberg-rust's Arrow predicate visitor evaluates null checks on list and map columns via arrow-native `is_null`/`is_not_null`; only struct columns are unsupported (`project_column` rejects `DataType::Struct`). The check therefore produces false positives on a very common pattern: Spark pushes `isnotnull(arr)` below `Generate` for `explode(arr)`, which declined whole scans and cascaded into JVM-side aggregate fallbacks. ## What changes are included in this PR? - `CometScanRule`: the complex-null-check scan predicate now declines only **struct**-typed columns; list and map columns are admitted. Comment and fallback message updated to describe the actual iceberg-rust limitation (`project_column`). - Removed the now-unused `isComplexType` import. ## How are these changes tested? - `CometScanRuleSuite` passes unchanged. - End-to-end validation on a 24-query derived TPC-H benchmark (Spark 4.1.3, Iceberg hadoop catalog, nested `array<struct>` table, SF1, 3 runs, cold caches): eliminating the list false-positives removes 123 scan declines and restores native execution — Comet/Iceberg total 20.63 s → 16.93 s (−17.9%), with parity vs vanilla Spark verified on all queries (details in #5731). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
