LuciferYang opened a new pull request, #13350: URL: https://github.com/apache/gluten/pull/13350
## What changes are proposed in this pull request? `FlushableHashAggregateRule` keeps the PartialMerge aggregate directly below the distinct-partial aggregate of Spark's one-distinct plan regular, because flushing it emits the same grouping and distinct key more than once. It only looked for an offloaded distinct-partial aggregate. When another DISTINCT aggregate on the same column is not offloaded, for example `histogram_numeric`, the distinct-partial aggregate runs in vanilla Spark, the offloaded PartialMerge aggregate below it becomes flushable, and `count(distinct ...)` counts the repeated keys. This matches the distinct-partial aggregate as any `BaseAggregateExec`, and looks through the local sort that Spark puts below a `SortAggregateExec`. If the query has only DISTINCT aggregates, the PartialMerge aggregate has no aggregate expressions and the grouping-only check already keeps it regular, so the bug needs at least one non-distinct aggregate in the query. ## How was this patch tested? Added a test to `VeloxAggregateFunctionsFlushSuite`. It runs `count(distinct z)`, `count(*)` and `histogram_numeric(distinct z, 5)` with `spark.sql.execution.useObjectHashAggregateExec` on and off, compares the result with vanilla Spark, and checks that the PartialMerge aggregate below the vanilla distinct-partial aggregate is not flushable. It also runs with columnar sort disabled, so the local sort stays a vanilla `SortExec`. It fails without this change, and also fails if the sort look-through is removed. The existing count distinct test now also has a non-distinct `count(*)`, so it covers the offloaded distinct-partial case instead of passing through the grouping-only check. ## Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Opus. Related issue: #13349. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
