rluvaton commented on PR #26116: URL: https://github.com/apache/datafusion/pull/26116#issuecomment-6746798735
I raised some perf concerned in: - https://github.com/apache/datafusion/issues/24704#issuecomment-6722003741 Reposting it here to avoid it being missed > **The partial-stage approach can cause huge perf problems even when there is enough memory**, because it stops aggregating effectively. > > Consider: > ```sql > SELECT x, sum(y) > FROM tbl > GROUP BY x > ``` > > - `batch_size` = 1K > - input rows = 1M > - unique groups = 10K > - enough memory to hold all 10K groups (even 100K) > > Say `x` cycles through `0..10K` repeatedly (A uniform distribution behaves almost the same). Every 1K input rows then contain 1K distinct groups, so the threshold is hit after every input batch and 1K groups are emitted. > > Result: `input_rows / batch_size` = **1000 batches (1M rows)** emitted, instead of `unique_groups / batch_size` = **10 batches (10K rows)**. > > Each emitted group holds a single input row, so the partial aggregation does no reduction at all. We pay for hashing and still send 100x more rows downstream. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
