LiaCastaneda commented on issue #22758: URL: https://github.com/apache/datafusion/issues/22758#issuecomment-4905207618
I'm not super familiar with the Allocator API, but it looks like they've already experimented with integrating it into arrow-rs -- https://github.com/apache/arrow-rs/pull/6336. Reading through that PR, I think the main blockers that led to closing it were that Allocator isn't compatible with dyn structs, so Buffer/Bytes can't carry a custom allocator, which they'd need to, since whatever allocated the memory has to free it on drop. Another concern was the fact that the allocator would have to be threaded up through `Buffer`, `Array`, and every kernel. Another approach suggested [here](https://github.com/apache/arrow-rs/issues/8938#issuecomment-3739770880) that avoids introducing breaking changes in the arrow apis was to build wrapper APIs in DataFusion around the arrow kernels that account for the memory of the arrays they create. It's still less accurate than a real Allocator (it claims memory after the allocation happens and can miss a kernel's intermediate buffers), but it's fully scoped to DataFusion and needs no arrow changes. I think it could work as a temporary solution until Allocator is stable and mature, although the arrow API breakage concern still stands whenever we revisit the real fix with Allocator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
