comphead commented on PR #6029: URL: https://github.com/apache/datafusion-comet/pull/6029#issuecomment-5739698492
Comet native allocations are not connected to overhead mem or offheap. The allocator just takes mem from pod OS not asking Spark. So Spark have no idea someone else like Comet is consuming memory, as well as Comet doesnt’ respect Spark memory params. Currently they just 2 neighbours living together in the same house, but not knowing about each other The only thing that likely unite then is CometTaskMemoryManager when offheap enabled, when disabled Comet just takes whatever it wants. I think we need to make this clarification in diagram, my head is struggling to come up with idea -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
