zhengruifeng opened a new pull request, #57969: URL: https://github.com/apache/spark/pull/57969
### What changes were proposed in this pull request? Use NaN as the uninitialized min/max sentinel in ml.stat.SummarizerBuffer and mllib.stat.MultivariateOnlineSummarizer. The add and merge paths now select non-NaN values over the sentinel, while implicit zeros still participate in min/max. Existing behavior that ignores NaN when a valid value or zero exists is preserved. ### Why are the changes needed? SPARK-20711 reports incorrect min/max results caused by using Double.MinValue and Double.MaxValue as initialization sentinels. Those values are valid inputs, and an all-NaN dimension can expose the numeric sentinel as its result. Using NaN distinguishes an uninitialized dimension from every valid numeric value. An all-NaN dimension now returns NaN. ### Does this PR introduce _any_ user-facing change? Yes. Min and max for a dimension containing only NaN values now return NaN instead of numeric initialization sentinels. Dimensions containing both NaN and non-NaN values retain the existing behavior of using the non-NaN values. ### How was this patch tested? Added coverage for all-NaN dimensions, NaN mixed with finite values and zero, Double.MinValue and Double.MaxValue, and both merge orders. The following suites and checks passed: - SummarizerSuite: 79 tests - MultivariateOnlineSummarizerSuite: 11 tests - MinMaxScalerSuite: 6 tests - mllib Scalastyle: zero errors and warnings ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
