andygrove commented on issue #587: URL: https://github.com/apache/arrow-datafusion/issues/587#issuecomment-894802884
We should also discuss creating a scheduler in DataFusion (see https://github.com/apache/arrow-datafusion/issues/64) since it is related to this work. Rather than try and run all the things at once, it would be better to schedule work based on the available resources (cores / memory). We would still need the ability to track/limit memory use within operators but the scheduler could be aware of this and only allocate tasks if there is memory budget available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
