rkrishn7 opened a new issue, #27149:
URL: https://github.com/apache/datafusion/issues/27149
### Is your feature request related to a problem or challenge?
`first_value` only provides a `GroupsAccumulator` when it has an `ORDER BY`:
```rust
fn groups_accumulator_supported(args: &AccumulatorArgs) -> bool {
!args.order_bys.is_empty() && matches!(/* supported types */)
}
```
Without an ordering, grouped aggregation falls back to
`GroupsAccumulatorAdapter`, which keeps one boxed
`TrivialFirstValueAccumulator` per group. Each of those holds its value as a
`ScalarValue`. Time and memory per group are therefore much higher than for the
native grouped accumulators, and the gap grows with the number of groups.
Unordered `first_value` is common because the planner produces it:
`ReplaceDistinctWithAggregate` rewrites `SELECT DISTINCT ON (keys) cols` with
no `ORDER BY` into an aggregate grouped by `keys` with one `first_value(col)`
per selected column. Any `DISTINCT ON` over a high-cardinality key takes this
path, as does `first_value(col)` written without `ORDER BY`. (Plain `SELECT
DISTINCT` is unaffected; it becomes a `GROUP BY` with no aggregates.)
On the ClickBench `hits` dataset, for example:
```sql
SELECT DISTINCT ON ("UserID") "UserID", "URL", "Title", "SearchPhrase",
"EventTime", "RegionID"
FROM hits;
```
There are 17.6 million distinct `UserID` values. On `main` (b6b2e0e), on a
14-core M4 Pro with 24 GB of memory, this query takes 73 s and the process
peaks at about 21 GB. A single `first_value("WatchID")` grouped by `UserID`
takes 6.2 s and peaks at about 6.6 GB.
### Describe the solution you'd like
A native `GroupsAccumulator` for `first_value` without `ORDER BY`. It needs
no ordering state: a group keeps the first qualifying row it sees and is never
written again. The existing `ValueState` implementations
(`PrimitiveValueState`, `BytesValueState`, `GenericValueState`) already
store values per group for the ordered accumulator and can be reused.
The intermediate state should stay `[first_value, is_set]`, the layout of
`TrivialFirstValueAccumulator`, so partial and final aggregation see the same
schema as today.
### Describe alternatives you've considered
- #25123 removed the adapter's per-batch walk over every group, which helps
this case too. Per-group cost (one boxed accumulator and a `ScalarValue` per
group) and memory remain.
### Additional context
`last_value` without `ORDER BY` has the same fallback. It could get a
similar accumulator but is left out of scope here; the planner does not produce
it.
### Willingness to contribute
I plan to contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
