sunchao opened a new pull request, #24804: URL: https://github.com/apache/datafusion/pull/24804
## Why are the changes needed? Parquet queries can incorrectly discard rows containing NaNs, or skip exact filtering for row groups that do not fully match. Parquet floating-point bounds omit NaNs, whereas DataFusion needs bounds over all non-null values in Arrow comparison order. For example, `[1.0, NaN]` can have bounds `[1.0, 1.0]` even though the NaN matches an ordered comparison or an IN predicate. Negative NaNs also invalidate lower bounds. ## What changes were proposed in this PR? - Extend the existing Parquet statistics-order guard to FLOAT, DOUBLE, and logical FLOAT16. File, row-group, and page min/max statistics become unknown; fully-matched proofs use the same guarded statistics. - Add the original SQL reproducer and real Parquet regressions for signed NaN payloads, nulls, IN/NOT IN caps, comparisons, and each pruning level independently. Verify null-count and integer/string pruning remain available. - Update existing float-pruning expectations and regenerate the affected SQL statistics snapshots. ## How was this PR tested? - Rust 1.97.0 and the upstream Arrow/Parquet 59.2.0 dependencies, based on `61bf6b96cc07d6a0518014dec7fe86fbef76576e`. - All three new Rust regressions fail with the unmodified upstream statistics guard and pass with this fix; all 253 Parquet crate unit tests pass. - Extended workspace tests with `avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`: 10,983 Rust tests passed, 8 ignored, and all 505 SQL test files passed. The initial run hit the host's 1,024-descriptor limit in an unrelated sort fuzz test; the successful rerun used a process-local limit of 65,536. - `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and the complete `./dev/rust_lint.sh` suite passed. The lint run used the upstream-pinned HawkEye 7.0.0. - Ran all 27 `parquet_metadata_statistics` benchmark cases before and after the fix in the unoptimized CI profile. These short measurements included slower no-statistics controls and do not establish query performance or absence of regressions. ## Which issue does this PR close? Closes #15812. ## Are there any user-facing changes? Min/max pruning based on Parquet floating-point metadata is conservatively unavailable, even for files that happen to contain no NaNs, because the metadata cannot prove NaNs absent. This may increase scanning and reduce selectivity-estimate precision. Exact row filtering and existing null-count and non-floating statistics behavior are unchanged. No public API or configuration changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
