mkleen commented on PR #26000:
URL: https://github.com/apache/datafusion/pull/26000#issuecomment-6943133242

   @alamb fair question. The earlier numbers were for an older version. I 
re-ran the current pr using the #24929 method (target_partitions = 1, 4M rows, 
~1M distinct x, smallest -m that finishes with -d 0 --mem-pool-type greedy).
   
   
   ```
   SELECT g, sum(DISTINCT x) FROM t GROUP BY g, with 
single_distinct_prefer_native off and on:
   
   
┌─────────┬──────────┬─────────┬──────────────┬─────────────┬───────────────┬──────────────┐
   │ Groups  │ Time off │ Time on │ Peak RSS off │ Peak RSS on │ Min limit off 
│ Min limit on │
   
├─────────┼──────────┼─────────┼──────────────┼─────────────┼───────────────┼──────────────┤
   │ 2,000   │ 0.124 s  │ 0.245 s │ 539 MiB      │ 306 MiB     │ 192M          
│ 80M          │
   
├─────────┼──────────┼─────────┼──────────────┼─────────────┼───────────────┼──────────────┤
   │ 500,000 │ 0.157 s  │ 0.405 s │ 538 MiB      │ 425 MiB     │ 192M          
│ 240M         │
   
└─────────┴──────────┴─────────┴──────────────┴─────────────┴───────────────┴──────────────┘
   ```
   
   In the current state, we are trading performance for memory efficiency when 
we have few large groups and no memory limit. At the same time it hurts with 
many small groups. So it's complicated and as @asolimando suggested in 
https://github.com/apache/datafusion/pull/26000#issuecomment-6265078225 this 
might be a cost-based-optimization (CBO) usecase. The current knob behaviour is 
tricky to figure out correctly.
   
   Anyway, your suggestion to makes things faster makes sense to me. So to to 
improve the performance of `sum(DISTINCT x)` maybe we should have a native 
distinct GroupsAccumulator for sum?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to