viirya opened a new issue, #26781: URL: https://github.com/apache/datafusion/issues/26781
### Describe the bug #26726 (for #26725) releases the capacity that a clustered aggregate table keeps after it emits its completed groups under memory pressure, but only for the group values and `group_indices`. Accumulators can also keep per-batch scratch, and `GroupsAccumulator` has no way to release it. `HllGroupsAccumulator` (`APPROX_DISTINCT`) keeps a `hashes: Vec<u64>` sized to the largest input batch. `size()` counts it, and `state` does not release it: https://github.com/apache/datafusion/blob/c18e5d974c49382ba892fa9f89ce71d3aa0ab8ae/datafusion/functions-aggregate/src/approx_distinct.rs#L593-L605 https://github.com/apache/datafusion/blob/c18e5d974c49382ba892fa9f89ce71d3aa0ab8ae/datafusion/functions-aggregate/src/approx_distinct.rs#L640-L644 With 8192-row batches, each `APPROX_DISTINCT` keeps 64 KiB after the emit. As reported by @sunchao in https://github.com/apache/datafusion/pull/26726, a fully clustered partial aggregate with `APPROX_DISTINCT(k)` and `APPROX_DISTINCT(v)` over one sorted batch of 8192 distinct `Int64` keys still fails a 100 KiB limit, although the table holds only the active group. ### To Reproduce A partial `APPROX_DISTINCT(k), APPROX_DISTINCT(v) GROUP BY k` over input sorted on `k`, with 8192 distinct keys per batch and a 100 KiB `GreedyMemoryPool`. ### Expected behavior After the memory-pressure emit, the reservation should cover only the active group's state, so the stream continues. ### Additional context One option: add a `GroupsAccumulator::shrink_to_fit` that does nothing by default, call it from `ClusteredAggregateTable::shrink_to_fit`, and implement it for accumulators that keep per-batch scratch, starting with `HllGroupsAccumulator`. ### Willingness to contribute I plan to contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
