2010YOUY01 commented on issue #17334: URL: https://github.com/apache/datafusion/issues/17334#issuecomment-3447981059
> What I would propose is that non-spillable operators _register_ their memory usage but don't error if they blow past the memory limit. That would put pressure on spoilable operators to start spilling without running into these sort of memory reservation deadlocks. I think this is what [@milenkovicm](https://github.com/milenkovicm) means as well by "we do not limit memory for non-spillable operators [...] Actually, we do track the memory usage of them, but we let it grow unbounded.". Probably we can let non-spillables go beyond a soft memory limit, but fails under a hard limit. Here are the rationales: Now non-spillable memory reservations are used in the operators that can use unbounded memory, but haven't implemented a spilling yet, and it's possible to implement spilling in the future. This approach works for the cases that the non-spillable operators can finish within the configured memory pool size, however now they're failing due to the current `FairSpillPool`'s problematic design under race conditions. This approach does not work if those non-spillable operators grow memory way beyond the configuration. All physical memory will be consumed before failure. Probably not an issue if we only run an query instance inside a isolated environment like docker, but it removes the opportunity for multiple concurrent queries to be registered in the same memory pool, which would enable more efficient memory sharing. An alternative I can think of is setting a soft memory limit with a hard memory limit (for example under 8GB soft memory limit, non-spillables are safe to go beyond it, and if it happens, spillables will have some response time to do the spill writes and free up memory; if they go beyond the hard memory limit -- let's say soft-limit * 120% -- the query fails. Another approach is letting spillables and non-spillables execute cooperatively -- if non-spillable OOMed, it will wait for spillables to do the spill writes, and wake up retry. But I think it would be very hard to implement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
