zhli1142015 commented on issue #10932: URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3442268927
@zhztheplayer my understanding is that when DynamicOffHeapSizingMemoryTarget is enabled, the following change causes the memory allocated by Velox not to be reported to the Spark Task Memory Manager, so spilling is not triggered — because DynamicOffHeapSizingMemoryTarget does not call the memoryTarget below. (Please correct me if I'm wrong.) <img width="2470" height="274" alt="Image" src="https://github.com/user-attachments/assets/2eb943e7-7936-4f69-9443-aaa900f6c8b5"; /> Even if the memory used by Velox is counted as on-heap memory, we still need to report the memory allocations to the Spark Task Memory Manager. Why this is not discovered when doing PR #9585 ? We’ve only tested PR #9585 extensively internally, but we are still using Gluten 1.3 internally, and RP #9336 has never been included. Spilling has not been an issue in our internal environment. I didn’t realize this issue until @wForget pointed it out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
