rluvaton commented on issue #22758: URL: https://github.com/apache/datafusion/issues/22758#issuecomment-4961567919
I'll give my 2 cents as we saw a couple of OOM and memory problem in our company with datafusion. we became very sensitive regarding memory so we started creating tests that check each operator in isolation with different data profile (number of columns, size of each row, etc.) and stress those in term of memory. the assertions on memory is by using custom allocator and checking if the **maximum** allocated memory across the entire run is over the limit by some config. we found several issues already in the few operators we started testing, I believe we will see much more. being conservative about memory and having good performance is a fine balance, for example, how can you know if you can preallocate an entire batch size values for each column in `RepartitionExec` impl if you have a lot of partitions or there is a lot of skew. -------- > I agree, ideally this would live in Arrow, although the community has expressed concern over how breaking it would be to integrate a `claim()` API into each Arrow buffer creation call and/or to adopt the still-unstable Allocator API. It looks like the main concern is that we shouldn't leave this in a mid-state, and that we should get it right the first time, given how breaking a change like this would be to public APIs like kernels, constructors, etc. > > I wonder if in the medium term, it would make sense to come up with a solution in DataFusion that doesn't require changes in arrow-rs, at least until we're able to use the rust Allocator (such as `claim`, even if it's not 100% accurate since it reports after allocation) -- which would be the permanent approach. @LiaCastaneda There are inherent problems with tracking allocation with arrow memory pool feature because the batches can have shared buffers and with arrow memory pool you avoid double counting memory, no operator is owning that batch if couple are holding it, which means that having one operator spill that batch may not free the memory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
