jayzhan211 commented on PR #25724: URL: https://github.com/apache/datafusion/pull/25724#issuecomment-6887611473
I'm turning this PR back to draft: with buckets on, it makes some aggregations faster and others slower, and I don't see a fix yet that doesn't add more tuned limits. On the benchmark runner (same commit, threshold 0 vs 262144): - Faster: ClickBench Q32 1.36x, Q15 1.23x, Q4 1.17x ([results](https://github.com/apache/datafusion/pull/25724#issuecomment-6887378076)); suite total 1.7%. TPC-H SF10 unchanged. - Slower: `clickbench_extended` Q15 (`covar_samp` grouped by `UserID`) 1.11-1.13x, and that suite's peak memory goes from 12.8 to 14.1 GiB ([results](https://github.com/apache/datafusion/pull/25724#issuecomment-6887429476)). My laptop reproduces it at 1.22x. My laptop also shows costs the description understates: - Peak memory up to 1.9x, not 1.13x, when the partial aggregation is skipped and groups repeat (4M groups, 8 aggregates): rows wait in buckets until they are compacted. - A query that ends just above the threshold pays for the move and gains nothing: ClickBench Q30 on half the data ends at about 264k groups per partition and runs 1.09x slower. - Under a 2 GB memory limit, the 8-aggregate query above runs 2.2x slower. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
