jiangxt2 opened a new pull request, #58030: URL: https://github.com/apache/spark/pull/58030
### What changes were proposed in this pull request? For `corr(x, y)` over n >= 2 valid pairs where either column has exactly zero variance (for example a constant column), the Pearson correlation is undefined, but the aggregate previously still evaluated `ck / sqrt(xMk * yMk)` with a zero denominator, which threw DIVIDE_BY_ZERO under ANSI mode. This PR adds a guard in `Corr.evaluateExpression` that returns an explicit `DoubleType` NULL before the division when `xMk` or `yMk` is exactly 0.0 and n >= 2. The n == 1.0 branch is kept ahead of the guard, so the legacy single-pair result is preserved: NaN with `spark.sql.legacy.statisticalAggregate` enabled, NULL otherwise. Valid zero-correlation results (ck == 0 while both variances are non-zero) still return 0.0. ### Why are the changes needed? For corr, a zero-variance result is undefined and should be represented as SQL NULL instead of failing in ANSI mode or depending on the division-by-zero fallback. Similar degenerate-input handling already exists in related statistical aggregates: for example, `regr_r2` checks the variance accumulators before dividing. See [SPARK-58213](https://issues.apache.org/jira/browse/SPARK-58213). ### Does this PR introduce _any_ user-facing change? Yes. For n >= 2 pairs where either column is constant, `corr` now always returns NULL: in ANSI mode this previously threw DIVIDE_BY_ZERO, and in non-ANSI mode the same NULL was produced implicitly through division by zero. The n == 1.0 single-pair behavior is unchanged. The primitive `DataFrameStatFunctions.corr` and MLlib `Statistics.corr` NaN contracts are not affected. ### How was this patch tested? Added a DataFrameAggregateSuite regression test covering constant x, constant y, both constant, all-NULL, partial NULL, single pair, normal and zero-correlation inputs, cross-checked across ANSI/non-ANSI and legacy/non-legacy configurations with whole-stage codegen on and off. The affected linear-regression, window and udf-window SQL golden files were regenerated by targeted SQLQueryTestSuite runs. The existing DataFrameWindowFunctionsSuite and DataFrameStatSuite pass with no test source changes. The targeted suites passed. The full SQLQueryTestSuite was not completed because an unchanged baseline identifier-clause.sql test stalled. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex and Claude AI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
