alamb commented on PR #4791: URL: https://github.com/apache/datafusion-comet/pull/4791#issuecomment-4909647996
> We should check with upstream datafusion. @alamb mentioned that the group by stuff is being rewritten to be more efficient and avoid this giant interim batch in the first place. I'd like to make sure if we tackle this in Comet it's a generalizable solution and not special-casing a hack. Also at a glance the comments seem huge and redundant. FWIW we are working on refactoring aggregates here - https://github.com/apache/datafusion/issues/22710 Then I think we will be in position to update the allocation strategy internally (which currently uses large contiguous allocations for group values and aggregates -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
