sunchao commented on code in PR #57460:
URL: https://github.com/apache/spark/pull/57460#discussion_r3639750776
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:
##########
@@ -589,6 +589,59 @@ class DataFrameAggregateSuite extends SharedSparkSession
Row(null, null, null, null, null))
}
+ test("SPARK-58291: empty-buffer merge must not overflow to NaN for
statistical aggregates") {
+ // A single-partition group with two equal, very large finite values has
zero variance, so
+ // var_pop / covar_pop / regr_sxy must be 0.0. Previously, when adjacent
Partial/Final
+ // aggregates were NOT combined (the old default), the Final merge of the
non-empty Partial
+ // buffer into the empty Final buffer computed `delta * deltaN * n1 * n2`
where `n1 == 0`;
+ // `delta * deltaN` overflowed to Infinity and `Infinity * 0 = NaN`,
corrupting the moments.
+ // CombineAdjacentAggregation (Complete mode) sidesteps the merge and
returned 0.0, so the two
+ // configurations disagreed. The merge fix makes both paths return 0.0.
+ // This must hold with and without AQE, and with combining on and off.
+ Seq(true, false).foreach { aqe =>
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString) {
+ val df = Seq(1e155, 1e155).toDF("a").repartition(1)
Review Comment:
[P3][non-blocking] Consider directly exercising both empty-buffer merge
directions.
These end-to-end cases reproduce the empty-left final-merge failure, but
`repartition(1)` does not independently exercise the newly added `isEmptyRight`
branch. A focused Catalyst regression using
`DeclarativeAggregateEvaluator.merge` with a populated buffer and an empty
buffer in both orders, ideally under both interpreted and generated execution,
would protect both branches without changing the production fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
