Rachelint commented on issue #24704: URL: https://github.com/apache/datafusion/issues/24704#issuecomment-5477459357
@rluvaton I think we should process the sorted case and the normal case(unsorted) in different ways like what is doing currently: # 1. For the sorted case Maybe we can continue to use `Emit::First(n)`. And if we decide to remove the `Emit::First(n)` finally (due to always unacceptable performance, and easy to be misused), I think we can make the `GroupValues` sorted input aware and use `Emit::All` to replace `Emit::First(n)`. The logic can be: - Assume the `group values` keeping `a`, and it is aware of the input is sorting, and has the api to return `current key` - When following batch `a, b, b, c` comes, we get `current key` from `GroupValues` first - Compare the current key `a` with `a,b,b,c`， and found the batch will lead to key switching - Split the the batch to `a` and `b,b,c`，and only put `a` into `GroupValues` - Call `Emit::All` and return the freezed `a` group - Input the `b,b,c` to reset `GroupValues`, and go to next loop # 2. For the unsorted case After many experiment, I found blocked approach is only a better memory management approach to reduce the peak memory usage. And I think our main target just be: - Support really return aggr state block by block, rather than holding a large guy + slicing - Don't lead to unacceptable performance regression I think the totally blocked approach #15591 may be not worthy: - Really large code changes - A massive impact on the existing architecture So after struggling with #15591 , I think something simple like `Emit::Last` may be a way more worthy trying. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
