xinghuayu007 opened a new issue, #10388: URL: https://github.com/apache/incubator-gluten/issues/10388
### Description GLUTEN-8018 increases heap memory when the fallback ratio over spark.gluten.auto.adjustStageResources.fallenNode.ratio.threshold may cause high memory consumption. For example, 10G for heap memory, and 15G for off-heap memory, using the follow setting. ' spark.gluten.auto.adjustStageResource.enabled=true; spark.gluten.auto.adjustStageResources.heap.ratio=3.0d; spark.gluten.auto.adjustStageResources.fallenNode.ratio.threshold=0.1d; ' When this rule is applied, heap memory will become 30G, and the total memory for gluten is 45G, that brings 20G consumption and causes OOM. In my ideal, it's better to decrease off-heap memory at the same time when increasing heap memory to make the total memory consumption no-increasing. ### Gluten version None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
