milenkovicm commented on issue #3941: URL: https://github.com/apache/arrow-datafusion/issues/3941#issuecomment-1289320935
Should memory limit be optimistic? What I mean in case of aggregation we could first process record batch, compare memory before and after batch is process and request delta value from memory manager. Otherwise we'd need to do two passes over records batch, or request memory for every record, which may lead to contention on memory manager and trigger spill in middle of batch processing. End of batch processing would be a "safe point" which should have correct memory usage, or trigger spill. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
