zhztheplayer commented on issue #10932: URL: https://github.com/apache/incubator-gluten/issues/10932#issuecomment-3437772147
The dynamic off-heap sizing code, before and after #9336, both don't trigger spill. Before #9336, DynamicOffHeapSizingMemoryTarget essentially caps the available off-heap size within a number which is lower than the Spark off-heap size because of the co-accounted on-heap memory in this target. Because this number is lower than the Spark off-heap size, and when the number is reached, OOM will be reported. So from Spark memory manager's view, it's used off-heap memory never reaches the total off-heap limit. That's why spill was never triggered. #9336 just simplified the code, so this spill shortage can also exposed to code reader clearly. I had a [similar question](https://github.com/apache/incubator-gluten/pull/9585#issuecomment-2909353000) about the spill design in @zhli1142015's later rework of this feature, and he may have more insights. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
