mohitgurav20 opened a new pull request, #26000: URL: https://github.com/apache/datafusion/pull/26000
## Which issue does this PR close? Closes #25932 ## Rationale for this change The `SingleDistinctToGroupBy` rule rewrites queries like `SELECT g, sum(DISTINCT x) FROM t GROUP BY g` into a two-step aggregation. While this rewrite can sometimes be faster, it can also consume over 2x more memory because the first step groups by both `g` and `x`. DataFusion now has highly optimized `GroupsAccumulator` implementations for aggregates like `sum`, `min`, and `max`. If an aggregate natively supports `GroupsAccumulator`, the two-step rewrite is unnecessary and highly memory-inefficient. Previously, the optimizer rule only checked if the rewrite was "worth it" if there was a `count(*)` in the query. If `count(*)` was absent, it blindly performed the memory-intensive rewrite. ## What changes are included in this PR? - Renamed `rewrite_pays_for_count` to `rewrite_pays`. - Modified `is_single_distinct_agg` to apply the `rewrite_pays` check **unconditionally** for all queries, rather than gating it behind the presence of a `count(*)` rollup. - Updated optimizer snapshot tests using `cargo insta accept`. As expected, many single distinct aggregation tests now produce simpler, single-step aggregation plans since the rewrite correctly bails out when the underlying aggregate supports `GroupsAccumulator`. ## Are these changes tested? Yes, the existing optimizer test suite validates these changes (with snapshots updated to reflect the new, more efficient plans for natively supported distinct aggregates). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
