LuciferYang opened a new issue, #13340: URL: https://github.com/apache/gluten/issues/13340
When regr_r2 runs one aggregation phase in Velox and another in vanilla Spark, it returns wrong results. This happens, for example, when the query also has a DISTINCT aggregate that Gluten does not offload: ```sql select k, regr_r2(y, x), histogram_numeric(distinct z, 3) from t group by k ``` On the 14-row table used by the tests in the fix, Gluten returns 0.0208 and 0.1924 for two groups where vanilla Spark returns 0.5771 and 0.4386. `VeloxIntermediateData` maps RegrR2 with Corr's field order, because `RegrR2` extends `PearsonCorrelation`. Velox's regr_r2 has a different intermediate layout, (c2, count, m2X, meanX, meanY, m2Y) with X being the second argument. Spark's `RegrR2(y, x)` passes its arguments to `PearsonCorrelation(y, x)`, so Spark's `xAvg` and `xMk` describe y. With Corr's order, Spark's `xMk` receives Velox's m2X, `yMk` receives meanX and `yAvg` receives m2Y. When every phase runs in Velox, the mapping is applied in both directions and cancels out, so only mixed plans are affected. This issue was written with the assistance of AI (Claude Opus). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
