alamb commented on PR #11627: URL: https://github.com/apache/datafusion/pull/11627#issuecomment-2256297468
> DuckDB's aggregation [implementation](https://duckdb.org/2024/03/29/external-aggregation.html) won't completely abandon the hash table in partial aggregation. Instead, they set a limit like 10,000 entries in the partial aggregation hash table, when it gets full it will flush local HT to the final aggregation state, and start from an empty table in local partial aggregation. I think @Dandandan reported he tried a similar approach here and that it wasn't as effective as we had hoped: https://github.com/apache/datafusion/issues/6937#issuecomment-1681310199 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
