marcuslin123 opened a new pull request, #57701: URL: https://github.com/apache/spark/pull/57701
### What changes were proposed in this pull request? This PR makes cached `FloatType` and `DoubleType` column statistics use Catalyst's SQL floating-point ordering when collecting lower and upper bounds. It also initializes both bounds from the first non-null value. Unit coverage verifies NaN bounds for both float and double values in either input order, including leading nulls. An end-to-end regression test verifies that cached table pruning retains rows that match a NaN filter. ### Why are the changes needed? The cached column statistics collector previously used JVM `<` and `>` comparisons. Every such comparison with NaN is false, so NaN could be omitted from a cached batch's upper bound. Cache pruning uses Catalyst ordering, where NaN sorts after finite values, and could therefore incorrectly prune a batch containing a matching NaN row. ### Does this PR introduce _any_ user-facing change? Yes. Queries filtering a cached floating-point column for NaN no longer risk returning an empty or incomplete result because a matching cached batch was incorrectly pruned. ### How was this patch tested? Added regression coverage and ran: ``` build/sbt 'sql/testOnly org.apache.spark.sql.execution.columnar.ColumnStatsSuite' build/sbt 'sql/testOnly org.apache.spark.sql.execution.columnar.InMemoryColumnarQuerySuite' build/sbt sql/scalastyle sql/Test/scalastyle ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (used for code assistance). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
