milenkovicm commented on issue #17334: URL: https://github.com/apache/datafusion/issues/17334#issuecomment-3496272249
Maybe the simplest change to mimic some kind of cooperative spilling is to have a global state in `MemoryPool`, like a counter or timestamp, which would indicate when was the last time a consumer failed to grow allocation, lets call it "failed allocation counter". `MemorConsumer` apart from providing `can_spill` should provide `last_spill_counter` so if the counter is less than memory pool counter memory pool could fail to grow memory reservation and force consumer to spill. Implementations could make this logic customized, like spill all or spill until we get under 50% of reserved memory, or whatever ... This could be backward compatible change and we could get some kind of cooperative spilling without big impact, having a mutex state "failed allocation counter" and `MemorConsumer::advance_counter` which would be triggered by MemoryPool in case of failed allocation (instead of relying on memory consumer to advance it when it spills) I also believe we would need some kind of soft and hard memory, where unspillable consumers should get memory allocation if we're over soft limit but can trigger spill with incrementing "failed allocation counter" Lastly we would probably need some kind of grace period between "failed allocation counter" increment so it does not trigger too many unnecessary spills (not sure about this part) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
