rich7420 opened a new issue, #7588: URL: https://github.com/apache/datafusion-comet/issues/7588
### What is the problem the feature request solves? #7156 stops passing `<`, `<=`, `>` and `>=` residuals on `FLOAT` and `DOUBLE` columns to iceberg-rust. At the pinned revision (`b154603`), these can drop rows that Spark returns: - The row filter compares floats with `arrow_ord::cmp`, which uses the IEEE 754 total order (`f32::total_cmp`). That order puts negative NaNs below every number. Spark puts every NaN above all numbers. - Row-group pruning trusts a NaN min or max. parquet-mr 1.9 and earlier write one when a row group starts with NaN (apache/iceberg-rust#3494). apache/iceberg-rust#3502 fixes this, but it landed after the pinned revision. - `PageIndexEvaluator::visit_inequality` checks only the literal for NaN, not the page bounds. Page-index pruning may have the same gap. I haven't checked this against a real file. Results stay correct. Iceberg's planning still skips data files by their bounds. The filter above the scan applies the predicate. But for these filters the native scan now reads every row group and page of the files that remain. The same holds for any `AND` that contains one of them, since partial pushdown is still open in #5883. ### Describe the potential solution 1. Bump the iceberg-rust pin past apache/iceberg-rust#3502. 2. Check page-index pruning against NaN page bounds. If it has the same gap as #3494, fix it upstream. 3. Make the row filter order NaNs as Spark does. Otherwise, push these residuals for pruning only and leave the row filtering to the filter above the scan. 4. Remove the `LT | LT_EQ | GT | GT_EQ` case for `FloatType` and `DoubleType` in `CometIcebergNativeScan.icebergExprToProto`. Remove the matching line in the Iceberg guide too. Zero literals stay out of pushdown either way. The native reader does not treat `-0.0` as equal to `0.0` (#7136). Tests should compare the native scan's results with Spark's. Use one file with negative and positive NaNs, both zeros and ordinary values. Use small pages so that page pruning has something to skip. ### Additional context Requested in the review of #7156: https://github.com/apache/datafusion-comet/pull/7156#pullrequestreview-5830589758 ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
