2010YOUY01 commented on issue #20715: URL: https://github.com/apache/datafusion/issues/20715#issuecomment-4001386158
> * Partition Aware Scheduling: if there are multiple partitions across one or more active queries, we should ensure that we fairly allocate memory according to partitions > * High and Low memory watermarks: we should have two thresholds for our memory limits. The first is the low watermark: when things should _start_ to spill, and high watermark: when we should error out because we are at max memory and there is no way to continue without blowing out memory allocations > * Ability to "wait" for memory to be free: rather than error out immediately if there is no memory, have the ability to wait until memory is available. This would mean that we would need some sort of method to wait until memory is available. However: this is deadlock prone so we need to be careful, or have it as an opt-in. I.e, have the default behaviour to error out immediately, but allow users of datafusion the ability to wait around for a bit I have some questions: For point 1, what is the difference between the proposed change and the existing mechanism in the fair pool? For point 2, currently we only have a single watermark. What are the benefits of introducing two watermarks? I can see that this may make operators less likely to wait for memory as mentioned in point 3, but I’m wondering if there are other considerations as well. Also for point 3, I can confirm this is a bug in the current implementation, and it would be great to see it fixed in the new memory pool implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
