mohitgurav20 commented on PR #26000: URL: https://github.com/apache/datafusion/pull/26000#issuecomment-6262445810
Thanks @mkleen for the detailed measurements — this is very helpful. You're right that the trade-off is clear: lower memory, but \~2.5× slower on the first two queries when the two-step rewrite is skipped. I'll work on adding a config setting so users can choose between: - best memory (skip the rewrite when GroupsAccumulator is available) - best speed (keep the current rewrite behaviour) Something like `datafusion.optimizer.single_distinct_to_group_by` (or similar) that defaults to the current behaviour, with an option to prefer the lower-memory path. I'll update the PR with the setting + docs + tests. Let me know if you have a preferred name or default for the flag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
