yashdotrv opened a new pull request, #27129: URL: https://github.com/apache/datafusion/pull/27129
## Which issue does this PR close? Addresses #26705. ## Rationale for this change A partial aggregate can emit a state batch containing all its input values and then release its memory reservation. The final aggregate merges that batch before reserving memory for it. For aggregates such as `median` and `array_agg`, this leaves input-sized state unaccounted for by the memory pool. ## What changes are included in this PR? Final aggregates now reserve incoming state batches before merging them. The hash and clustered final paths can spill their current state and retry when the pool cannot admit a batch. I also updated the affected memory reservation and spill tests. PR #26728 adds related memory accounting tests, but its cases for this issue are currently ignored. ## What is the testing strategy for this PR? I added reservation tests for the ungrouped, hash grouped, and clustered grouped final aggregate paths. The extended workspace test suite, formatting check, and Clippy pass. I also ran the issue’s 30-million-row, four-partition `median` reproducer with a 300 MB memory limit. It now returns `ResourcesExhausted` when the incoming state cannot be reserved. Peak RSS can still exceed the configured limit; reducing temporary memory use during buffer growth is outside this PR. ## Are there any user-facing changes? Under memory pressure, a final aggregate may now spill or return `ResourcesExhausted` instead of continuing with unreserved state. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
