HairstonE opened a new pull request, #25390:
URL: https://github.com/apache/datafusion/pull/25390
## Which issue does this PR close?
- Closes #21051.
## Rationale for this change
Distinct accumulates all the values and then counts them, even if the
threshold is met it continues through the rest of the dataset.
Right now `count(distinct)` doesn't go to the distinct accumulator, DF
puts two
aggregates on top of each other (a group by and then a count of the
groups).
Measured on a 50M-row memory table (same binary, toggled via
`SET datafusion.optimizer.enable_distinct_aggregation_soft_limit`):
| Query | Optimization on | Optimization off |
|----------------------------------|-----------------|------------------|
| `count(DISTINCT col) > 1`, 5 distinct values | 0.007–0.012 s | 0.16–0.37
s |
| `count(DISTINCT col) > 1`, 50M distinct values | 0.006–0.013 s | 4.0–4.7
s |
## What changes are included in this PR?
`SELECT count(DISTINCT col) > 1 FROM table` type queries are when this code
runs. Take the constant value + 2, one extra for the NULL case and another
for
clearing the threshold. When we are counting the groups we track the count
until n + 2, if we hit that number we short circuit. This short circuit is
checked after each batch in Partial.
HAVING was left out for simplicity.
Both the existing `LIMIT` pushdown and the new comparison pattern
do the same thing so they share the flag
(`enable_distinct_aggregation_soft_limit`).
## What is the testing strategy for this PR?
Yes.
- count_distinct_comparison_sets_soft_limit
- count_distinct_escaping_count_prevents_soft_limit
- count_distinct_comparison_capped_matches_uncapped
- existing limited_distinct_aggregation tests
- sqllogictests: aggregate, limit, count, explain
## Are there any user-facing changes?
No user facing changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
