ding-young commented on issue #17334: URL: https://github.com/apache/datafusion/issues/17334#issuecomment-3393076808
@milenkovicm Thank you for sharing the experience. > The other reason is that we can’t do much with them; from our experience, they freak out a bit too early before a spill can occur. Yes, when a non-spillable operator runs out of memory, it’s difficult to trigger spilling in another spillable operator to reclaim memory which seems to be a limitation currently. Have you ever observed any cases where a non-spillable operator showed a memory usage spike (for example, due to skewness or similar factors)? I wonder what would be the solution for these frequent failures on non-spillable operators - especially when other concurrent operators are spillable. If the memory usage of non-spillable operators can be roughly estimated before execution, do you think it would make sense to bypass or pre-reserve memory for them, instead of continuously growing the shared memory reservation along the non-spillable path? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
