Rachelint commented on PR #15591: URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5463126272
Actually, the main goal of the blocked approach is to release aggregation state progressively while emitting output batches, instead of retaining all state until all batches have been produced. We could originally use EmitTo::First for this, but it is too expensive because removing the first groups requires shifting/copying the remaining state and renumbering group indices. We then explored organizing the state into blocks in this PR, but that introduces substantial implementation complexity. So perhaps a simpler alternative is to add EmitTo::Last. Removing the last groups would not require shifting the remaining state or changing its group indices, while still allowing us to release state progressively during final output. One caveat is that reducing Rust-visible Vec capacity does not necessarily mean RSS is released immediately. That depends on the allocator: jemalloc may split and purge the trailing pages of a large allocation, while mimalloc may retain the underlying allocation until it becomes significantly smaller. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
