jayzhan211 commented on code in PR #24961:
URL: https://github.com/apache/datafusion/pull/24961#discussion_r3943217573
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1327,35 +1343,27 @@ impl AggregateExec {
matches!(
self.mode,
AggregateMode::Single | AggregateMode::SinglePartitioned
- ) && self.limit_options.is_none()
- && self.input_order_mode == InputOrderMode::Linear
+ ) && self.input_order_mode == InputOrderMode::Linear
Review Comment:
Routing change here drops the distinct soft-limit optimization for
Single/SinglePartitioned aggregates.
On main, `should_use_single_hash_stream` required `limit_options.is_none()`,
so a distinct aggregate with `lim=[n]` fell through to the legacy
`GroupedHashAggregateStream`, which stops consuming input once `n` groups
exist. With this change it routes to `SingleHashAggregateStream`, which has no
`group_values_soft_limit` and reads the whole input. The plan text is unchanged
so no slt catches it, but e.g. `aggregate.slt:7290` (`mode=SinglePartitioned,
aggr=[], lim=[5]`) now scans everything under the limit. That is the shape
`LimitedDistinctAggregation` produces for `SELECT DISTINCT ... LIMIT n` on
single-partition / hash-partitioned inputs.
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1327,35 +1343,27 @@ impl AggregateExec {
matches!(
self.mode,
AggregateMode::Single | AggregateMode::SinglePartitioned
- ) && self.limit_options.is_none()
- && self.input_order_mode == InputOrderMode::Linear
+ ) && self.input_order_mode == InputOrderMode::Linear
Review Comment:
I think we might need `group_values_soft_limit` like
`PartialReduceHashAggregateStream` for `SingleHashAggregateStream`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
