rank-p opened a new pull request, #27290: URL: https://github.com/apache/datafusion/pull/27290
## Which issue does this PR close? Closes #15812. Related to #26490. ## Rationale for this change Parquet float min/max can exclude NaNs. Treating these as complete bounds can discard matching rows, remove necessary filters, return incorrect `MIN/MAX`, or replace `[1.0, NaN]` with `[1.0, 1.0]`. This draft is a count-aware alternative to #24804, which withholds all float bounds. It keeps bounds when the existing metadata proves they are safe. I am proposing it separately to discuss that distinction and the fallback cost before changing the pruning API. Related approaches are the closed #15821 and the narrower constant-column fix in #27283. ## What changes are included in this PR? - Validate float bounds when exposing file, row-group, and page statistics. - Require matching stored/output float types, IEEE total order, and proof of zero NaNs. Missing counts are unknown. Page counts take priority, with a known-zero group count covering missing page counts. - Withhold unsafe bounds while keeping unrelated statistics. No public API changes. Restoring more float range pruning is left for follow-up. ## What is the testing strategy for this PR? Regressions cover both NaN signs across Float16/32/64, missing/zero/positive counts, and independent row-group/page pruning. SQL and DataFrame tests cover returned values, metadata aggregates, signed zero, and mixed float schemas. Removing the order or type guard makes its query regression fail. On the updated base, 13,626 Rust tests passed (8 ignored), all 547 SQL files passed, and 116 CLI tests passed. Formatting, strict Clippy, the full lint suite, and the metadata benchmark smoke test also passed with Rust 1.99.1. ## Are there any user-facing changes? Queries return correct values, but affected min/max optimizations are lost for unknown/positive NaN counts, `TYPE_ORDER` floats (even with zero NaNs), and float schema casts. This cost can be large: on base `9d15cf079`, one selective query over an 8.4-million-row local file went from about **0.4 ms to 25.5 ms**, reading all 64 groups instead of one. The same data with IEEE order and zero NaN counts stayed near 0.4 ms. These are warm local-file measurements, not a general slowdown estimate. Feedback on this conservative fallback is the main reason for opening as a draft. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
