jayzhan211 commented on PR #25724:
URL: https://github.com/apache/datafusion/pull/25724#issuecomment-6887223356

   @comphead the conflicts are resolved. Could you run your 1TB benchmark with 
`datafusion.execution.hash_aggregate_bucket_threshold` set to 262144 and to 0? 
It is a session setting, so both runs can use the same build.
   
   What to expect:
   
   - Only aggregations whose group keys and state are fixed-width use buckets, 
and only with spilling enabled (a disk manager). String-keyed aggregations 
should be unchanged.
   - The gain grows with the groups per final partition. Below the threshold 
nothing changes; up to about twice the threshold the gain is small.
   - Peak memory is worth recording: when the partial aggregation is skipped 
and groups repeat, rows wait in buckets until they are compacted. On input with 
7.5 rows per group that was 1.13x of one table.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to