mkleen commented on PR #26000: URL: https://github.com/apache/datafusion/pull/26000#issuecomment-6262381492
Before we move forward, we need speed and memory measurements before and after this change. I ran the queries from #25932 before and after this PR, using this table: ```sql CREATE TABLE t AS SELECT v % 2000 AS g, (v * 48271) % 999983 AS x, v % 1000 AS y FROM (SELECT unnest(generate_series(0, 3999999)) AS v); ``` | Query | Memory needed | Memory used | Time | |---|---|---|---| | `SELECT g, sum(DISTINCT x) FROM t GROUP BY g` | 192M → 80M | 538 → 306 MiB | 0.12 → 0.30 s | | `SELECT g, sum(y), sum(DISTINCT x) FROM t GROUP BY g` | 224M → 80M | 599 → 306 MiB | 0.13 → 0.32 s | | `SELECT g, count(*), sum(y), sum(DISTINCT x) FROM t GROUP BY g` | 80M → 80M | 306 → 306 MiB | 0.30 → 0.32 s | Memory goes down, but the first two queries are about 2.5x slower. Since neither is better in every case, we most likely need a setting that lets users choose between best speed and best memory by turning this optimization on or off. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
