Revanth14 opened a new pull request, #2418: URL: https://github.com/apache/iceberg-go/pull/2418
## Problem `ExpressionEvaluator` decides `<`, `<=`, `>` and `>=` with `cmp.Compare`, which orders NaN below every number, so a NaN value matched `x < 5` and `x <= 5`, and failed `NOT(x < 5)`. The Arrow row filter a scan applies follows IEEE 754, where every ordering comparison with NaN is false, so the exported evaluator and a scan disagreed on NaN rows (#2359). ## Fix The four ordering comparisons in `exprEvaluator` return false for a NaN value before comparing. `=` and `!=` already gave the IEEE answer for a NaN value and are unchanged. This deliberately differs from Java's `Evaluator`, which uses `Double.compare` and orders NaN above every number; iceberg-go's evaluator follows its own row filter instead. ## Testing `TestExpressionEvaluatorMatchesRowFilterForNaN` runs each comparison, plain and under NOT, on float64 and float32 columns through both `iceberg.ExpressionEvaluator` and the row filter from `prepareBatchFilter`, over -Inf, 1, 5, 10, +Inf and NaN, and requires the same rows. On `main` the eight `<` and `<=` cases fail. NULL rows are left out on purpose: the evaluator is two-valued, so a comparison with NULL can't match the row filter's unknown result without three-valued evaluation. ## Context Closes #2359. Follow-up from #2355. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
