mkleen commented on PR #26000:
URL: https://github.com/apache/datafusion/pull/26000#issuecomment-6557742123
Here's a set of benchmark queries for the flag. They use the table from the
#25932,
plus string and float columns, so they cover integers vs. non-native
accumulators:
```sql
CREATE TABLE t AS
SELECT v % 2000 AS g, -- 2,000 groups
v % 10 AS g_few, -- 10 groups
(v * 48271) % 999983 AS x, -- ~1M distinct ints
CAST((v * 48271) % 999983 AS VARCHAR) AS s, -- same values as
strings
CAST((v * 48271) % 999983 AS DOUBLE) AS f, -- same values as
floats
CAST(v % 100 AS SMALLINT) AS si, -- small-int, 100
distinct
v % 1000 AS y
FROM (SELECT unnest(generate_series(0, 3999999)) AS v);
```
1. With GROUP BY, many groups (where the rewrite's memory cost shows most)
```sql
SELECT g, count(DISTINCT x) FROM t GROUP BY g;
SELECT g, sum(DISTINCT x) FROM t GROUP BY g;
SELECT g, count(DISTINCT s) FROM t GROUP BY g;
SELECT g, avg(DISTINCT f) FROM t GROUP BY g;
```
2. With GROUP BY, few groups (each group's distinct set is large)
```sql
SELECT g_few, count(DISTINCT x) FROM t GROUP BY g_few;
SELECT g_few, sum(DISTINCT x) FROM t GROUP BY g_few;
SELECT g_few, count(DISTINCT s) FROM t GROUP BY g_few;
```
3. With companion aggregates (the rule also rewrites these)
```sql
SELECT g, sum(y), sum(DISTINCT x) FROM t GROUP BY g;
SELECT g, count(*), count(DISTINCT x) FROM t GROUP BY g;
SELECT g, min(y), max(y), count(DISTINCT s) FROM t GROUP BY g;
```
4. No GROUP BY (here the rewrite is the only way to parallelise the distinct)
```sql
SELECT count(DISTINCT x) FROM t;
SELECT sum(DISTINCT x) FROM t;
SELECT count(DISTINCT s) FROM t;
SELECT count(DISTINCT si) FROM t; -- small-int case
SELECT count(*), count(DISTINCT x) FROM t;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
