alamb commented on issue #3941: URL: https://github.com/apache/arrow-datafusion/issues/3941#issuecomment-1290676317
> Should memory limit be optimistic? What I mean is that in case of aggregation we could first process record batch, compare memory before and after batch is process and request delta value from memory manager. Yes, I agree this would be best strategy: as memory is needed, we request more memory from the memory manager incrementally. > End of batch processing would be a "safe point" which should have correct memory usage, or trigger spill. wdyt? Yes In general I think upgrading / improving the memory manager would likely be fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
