mohitgurav20 commented on PR #26000: URL: https://github.com/apache/datafusion/pull/26000#issuecomment-6265233756
@asolimando @mkleen Makes total sense regarding the lack of logical plan statistics—a config knob fits well here. I've pushed the update in commit `222ff4e` and merged the latest `main`: - Added `datafusion.optimizer.single_distinct_aggregation_prefer_memory` (defaults to `false`). - Default (`false`): Lone distinct queries keep the existing two-phase group-by rewrite so throughput and partition fan-out remain unchanged. - Opt-in (`true`): Skips the rewrite when functions natively support `GroupsAccumulator`, keeping intermediate memory low for workloads under pressure. - Queries that need the rewrite for correctness (e.g. `approx_median`) still rewrite regardless of the flag. I've also added unit tests for both flag states, updated the sqllogictests and information schema, and refreshed the config docs. Let me know what you think! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
