rich7420 opened a new pull request, #6847: URL: https://github.com/apache/datafusion-comet/pull/6847
## Which issue does this PR close? Closes #5883. ## Rationale for this change A filter such as `l IS NOT NULL AND id > 5` currently loses native Iceberg pruning on `id` when the collection null check cannot be pushed down. ## What changes are included in this PR? Retain supported predicates in positive conjunctions while preserving the exact post-scan filter. OR still requires a safe predicate from both branches; NOT requires its entire child to convert exactly. Add serializer and native scan regressions for complex null checks, nested AND/OR/NOT, result parity and reduced reader I/O. Update the Iceberg user guide. ## How are these changes tested? On main `bfacc43a5`, Spark 4.1 / JDK 21: - Root-reactor compilation, Spotless and Scalastyle passed. - 28 serializer tests passed. - Native Iceberg testing completed with 118 passing test executions and one expected cancellation for SPARK-55626. The new regression confirms Spark result parity, native execution and reduced bytes scanned. - Native library built from this checkout and verified against the library used by the JVM tests. Please apply `run-iceberg-tests` for broader Iceberg-version coverage; keep Draft pending CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
