andygrove commented on issue #5418: URL: https://github.com/apache/datafusion-comet/issues/5418#issuecomment-6192492552
I checked this against main at 66c63bd307 on Spark 4.1.3, and the repro no longer returns NULL. Since #5421, a Spark final aggregate keeps the AVG partial below it in Spark too, so the query returns 2.0. That holds with AQE on and off, with the final-mode switch, and with Comet shuffle disabled. 1.1.0 has this guard and branch-1.0 has it through #6422, but 1.0.0 shipped without it. The native state hasn't changed. An untouched scalar `AvgAccumulator` still exports `(NULL, 0)`. When I let AVG partials feed a Spark final again, the same query returns NULL in every mixed configuration, and the `CometAggregateSuite` test `COUNT and AVG fall back together across a Spark shuffle` catches it. Grouped AVG already exports `(0.0, 0)` and stays correct. So on main this now costs coverage, not correctness: AVG partials, along with the other aggregates in the same operator, run in Spark whenever the final does. #5420 fixes the state and lets nondecimal AVG partials feed a Spark final again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
