LuciferYang opened a new issue, #13349: URL: https://github.com/apache/gluten/issues/13349
With flushable aggregation enabled (the default), `count(distinct ...)` can return a value that is too large when the query also has a DISTINCT aggregate that Velox does not support, such as `histogram_numeric`, plus a non-distinct aggregate: ```sql select k, count(distinct z), count(*), histogram_numeric(distinct z, 5) from t group by k ``` On a 2.4M-row table with 2.24M distinct values of z, 8 shuffle partitions and default settings, vanilla Spark returns 746667 for `count(distinct z)` in group 0 and Gluten returns 798356. Spark plans a single DISTINCT as four aggregates. `FlushableHashAggregateRule` keeps the PartialMerge aggregate on (k, z) regular, because flushing it emits the same (k, z) key more than once and the distinct-partial aggregate above it then counts the key again. The rule only recognizes the distinct-partial aggregate when it is offloaded. Here `histogram_numeric` keeps the distinct-partial aggregate in vanilla Spark while the PartialMerge aggregate below it is offloaded, so the PartialMerge aggregate becomes flushable. Before #12098 the rule only rewrote aggregates reached from a shuffle exchange through projects, so it never reached this PartialMerge aggregate. The Bolt copy of the rule still works that way. This issue was written with the assistance of AI (Claude Opus). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
