yashaggarwal85d opened a new issue, #26777: URL: https://github.com/apache/datafusion/issues/26777
### Is your feature request related to a problem or challenge? This came up in review of #26637. With partial group clustering (input sorted on a prefix of the group keys), the clustered aggregate table keeps its allocated capacity after it emits completed groups with `EmitTo::First`. `ClusteredAggregateTable::memory_size()` doesn't go down, so the stream's memory reservation stays at its peak. Example: `SUM(v) GROUP BY a, b` over input sorted on `a` only (`GroupClusteringMode::Partial([0])`), in 6 batches of 256 rows each, with two batches per value of `a`. The breakdown below shows the table before and after emitting the completed groups of `a = 0`: | | groups | accumulators | group values | clustering | group indices | total | |---|---|---|---|---|---|---| | before emit | 512 | 4096 | 24912 | 136 | 2048 | 31192 | | after emit | 256 | 4096 | 24912 | 136 | 2048 | 31192 | Both the group values (`GroupValuesColumn<true>`) and the `SUM` accumulator keep the capacity they had for 512 groups. Single and Final mode show the same numbers. The capacity is reused for the next run of groups, so memory doesn't keep growing. The problem is that a clustered Final or Single aggregate without spilling (for example, with temporary files disabled) can't use emitting to get back under a tight limit. After #26637, the stream emits the completed groups when a reservation is refused, but the reservation it needs afterwards is the same size, so it still fails with `ResourcesExhausted`. When the input is sorted on all group keys (full group clustering), `GroupValuesClustered` does release memory on emit, so the same limit is met. ### Describe the solution you'd like After emitting completed groups under partial clustering, release capacity that the remaining groups don't need. That could mean shrinking the group values and accumulator buffers when the emitted prefix is a large share of the table, or tracking the reservation by the live groups rather than by capacity. Then a refused reservation could be met by emitting, as it already is under full clustering. ### Describe alternatives you've considered - Keep the current behavior. Capacity reuse avoids reallocating for every run of groups, and with temporary files enabled the stream spills instead, so this only matters when spilling is unavailable. - Shrink only on the out-of-memory path, so the common path keeps the reuse benefit. ### Additional context Found while working on #26637 (fix for #26628). @comphead suggested tracking it separately rather than locking the current behavior into a test there. ### Willingness to contribute None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
