zhztheplayer commented on issue #10932: URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3442435675
> We’ve only tested PR https://github.com/apache/incubator-gluten/pull/9585 extensively internally, but we are still using Gluten 1.3 internally, and RP https://github.com/apache/incubator-gluten/pull/9336 has never been included. Spilling has not been an issue in our internal environment. I didn’t realize this issue until @wForget pointed it out. Now it gets clearer. That's also why I also raised a concern at https://github.com/apache/incubator-gluten/pull/9585#issuecomment-2909353000. Only for a solution like #9585, passing the memoryTarget to `DynamicOffHeapSizingMemoryTarget` for spilling could potentially make sense. But again, as a non-blocker comment, perhaps more comprehensive consideration is needed to make the feature more robust, like how to trigger spill when on-heap memory already exceeds 40%, how to avoid process RSS overrun if there're constant on-heap memory allocations that DynamicOffHeapSizingMemoryTarget isn't aware, etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
