mohitgurav20 commented on PR #26000: URL: https://github.com/apache/datafusion/pull/26000#issuecomment-6578268474
Yes, we can definitely do that. From the benchmarks, the only cases that got worse were: 1. Queries without `GROUP BY` (because they lose multi-threading). 2. String columns (because of per-string memory allocations). So under the flag, we can only skip the rewrite if: - The query has a `GROUP BY` (`!aggr.group_expr.is_empty()`) - The distinct column is a fixed numeric/primitive type (not strings) That way, strings and queries without `GROUP BY` will continue using the 2-phase rewrite as normal, and only numeric `GROUP BY` queries will skip it—giving the 60%–76% memory savings with no real performance hit. I can push this change if that sounds good to you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
