alamb commented on issue #11523: URL: https://github.com/apache/datafusion/issues/11523#issuecomment-2260759076
@wiedld has an initial PR to add `TrackedMemoryPool` in https://github.com/apache/datafusion/pull/11665 Here is my proposal for what remains to close this issue - [ ] Improving the default error message - [ ] Using the `TrackConsumersPool` as the default memory pool # Thoughts on error messages Message today: > Failed to allocate additional {} bytes for {} with {} bytes already allocated - maximum available is {} @wiedld 's proposal on https://github.com/apache/datafusion/pull/11665/files#r1693465283 > Failed to allocate additional {} bytes for {} with {} bytes already allocated for this reservation - {} bytes remain available for the total pool I think the new proposal is better as it is clearer what is going on # Thoughts on changing the default pool The the default is set here: https://github.com/apache/datafusion/blob/89677ae66ec2238ffd58de8c36fe6030fad458fa/datafusion/execution/src/runtime_env.rs#L211 I believe we should change the default pool in DataFusion to be a `TrackedMemoryPool<GreedyMemoryPool>` otherwise the error messages when a memory limit will continue to have the issues that are described in this tickets. The only potential issue with changing the default is that the tracking has some additional runtime overhead -- therefore we should run benchmark tests to ensure there is no performance regression. Also, a nice part of @wiedld 's `TrackedMemoryPool` design is that someone finds the overhead to be too large, they can change back to the existing `GreedyMemoryPool` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
