Revanth14 commented on issue #2131: URL: https://github.com/apache/iceberg-go/issues/2131#issuecomment-6298130509
I checked Java for this. Its row `Evaluator` doesn't rewrite NOT: it binds the expression directly, and only the metrics evaluators call `rewriteNot`. It also compares doubles with `naturalOrder`, where NaN sorts above every number, so Java's row evaluator says `NaN >= 5` is true while its metrics evaluator prunes `[1, NaN, 2]` on the upper bound. Java has the same split, so it doesn't decide this one. Two ways to make rows and pruning agree: - **(a) Rewrite NOT before building the row filter**, as suggested above. Simple, but `NOT(x < 5)` then never matches NaN, unlike IEEE comparison semantics. - **(b) NaN-aware NOT rewrite for float/double**: `NOT(x < 5)` → `x >= 5 OR is_nan(x)`. This keeps `NOT(NaN < 5)` true, and pruning stays correct because the `is_nan` branch keeps files with NaN values. It needs the bound type, so it would apply after binding rather than in the exported `RewriteNotExpr`. I lean towards (b). @zeroshade @laskoviymishka which would you prefer? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
