alamb commented on issue #824: URL: https://github.com/apache/arrow-datafusion/issues/824#issuecomment-894776286
I am also very interested in the ability to both track and limit memory usage for a datafusion plan -- there is some additional discussion / context related to this on https://github.com/apache/arrow-datafusion/issues/587. @jorgecarleitao 's implementation idea in https://github.com/apache/arrow-datafusion/issues/824#issuecomment-894746647 sounds like a good start. I don't think we can realistically try and limit memory usage before we can measure it. As for the single global context, it is also my preference that the code that is using DataFusion to manage the contexts. So specifically it would make sense to me if `datafusion-cli` had a single global context that it then passed into the datafusion crate but not to add a global context into the `datafusion` crate itself -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
