rluvaton commented on issue #24704: URL: https://github.com/apache/datafusion/issues/24704#issuecomment-6722003741
@jayzhan211 sorry, missed the notification. thanks for the explanation. first of all I think if we are having a major change in the aggregate code and external API we should not be good in memory in some cases only. group by strings keys, structs, list, etc is one of the use cases as well as array_agg. > and neither PR changes single-stage aggregation (one input partition) what do you mean by that? `complete` aggregation? because this is valuable as well ---- **The partial-stage approach can cause huge perf problems even when there is enough memory**, because it stops aggregating effectively. Consider: ```sql SELECT x, sum(y) FROM tbl GROUP BY x ``` - `batch_size` = 1K - input rows = 1M - unique groups = 10K - enough memory to hold all 10K groups (even 100K) Say `x` cycles through `0..10K` repeatedly. Every 1K input rows then contain 1K distinct groups, so the threshold is hit after every input batch and 1K groups are emitted. Result: `input_rows / batch_size` = **1000 batches (1M rows)** emitted, instead of `unique_groups / batch_size` = **10 batches (10K rows)**. Each emitted group holds a single input row, so the partial aggregation does no reduction at all. We pay for hashing and still send 100x more rows downstream. A uniform distribution behaves almost the same (~950 batches). ---- > once the Final table outgrows the CPU caches, the stream moves its state into 64 hash buckets and sends the rest of the input there without aggregating it and you change the accumulators into states and then combine then back? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
