plusplusjiajia opened a new pull request, #10877: URL: https://github.com/apache/paimon/pull/10877
### Purpose Stats-based file pruning compared NaN min, max or literal values with IEEE semantics, where every comparison with NaN is false. For example, a file whose max is NaN was skipped for `greater_than`, so rows in that file were missing from the result. Java orders NaN above every other value (`CompareUtils`) and keeps such files, and Java's stats collectors can write NaN bounds, for example for Avro. Stats with a NaN min, max or literal now keep the file. This only makes pruning more conservative: such files are read and filtered row by row. PyPaimon's row-level comparisons stay IEEE, so the stats do not adopt Java's ordering yet. Full parity for NaN and signed zero is left for a separate change. ### Tests - `predicates_test.py#test_by_simple_stats_with_nan`: FLOAT and DOUBLE stats with a NaN min, max or literal keep the file for every comparison, `in`, `not_in`, `between` and `not_between`, and finite stats keep their results. - `predicates_test.py#test_append_read_with_nan_stats`: an Avro append table whose file has a NaN max, as Java's stats collector writes it, returns the matching row for `greater_than`. - 17 test cases fail on master. The predicate tests pass locally: 77 tests. The tests matching predicate, pushdown, filter, stats or pruning pass: 1540 passed, and the 6 failures also fail on master because torch, vortex or the Java e2e data are missing locally. flake8 is clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
