alamb commented on PR #24785: URL: https://github.com/apache/datafusion/pull/24785#issuecomment-5497952504
> The bigger issue, I believe, is that we have to keep the large output batch in memory for a long time until it's finally dropped. I'd suggest revisiting the implementation once blocked memory management ships, to make it more memory-efficient. Yes I agree -- we have lots of pent up demnd (and excitement) to help get blocked memory management in, so I expect we will handle it better in the future. I think our bar for merging these PRs should be "as good as the current implementation" (otherwise it might be seen as a regression) As long as that bar is satisfied we can/should merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
