qiaojizhen commented on PR #53383: URL: https://github.com/apache/spark/pull/53383#issuecomment-3839209483
> > @VindhyaG I have found another issue. If spark.memory.offHeap.enabled = true, the Driver actually won't apply the offHeap memory in any cluster deploy mode. But now the Driver is displaying the number which is set by spark.memory.offHeap.size. So I think the Driver in the web ui should display zero for "Off Heap Memory". Could u please solve this problem in this PR by the way？ Thanks! > > @qiaojizhen I did not get what you mean by Driver won't aply offheap in cluster mode. Is it not supported in cluster mode you mean? @VindhyaG Yes! The off-heap memory won't work for Driver in any cluster mode! If I use the following config： ``` spark.driver.memory=1g spark.driver.memoryOverhead=1g spark.memory.offHeap.enabled=true spark.memory.offHeap.size=2g ``` when I submit the job to YARN, the Driver container only gets 2G memory(including spark.driver.memory & spark.driver.memoryOverhead): <img width="452" height="244" alt="image" src="https://github.com/user-attachments/assets/e2f125e2-98f2-408e-8ec4-1b22cace0684"; /> but the web ui still display the off-heap memory: <img width="648" height="387" alt="image" src="https://github.com/user-attachments/assets/4da1b9c4-410e-4069-a708-7165c8fdd5d2"; /> and the soucre code here [Spark Yarn Client](https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala) has proved that it will only use spark.driver.memory & spark.driver.memoryOverhead when applying the AM memory: <img width="1586" height="747" alt="image" src="https://github.com/user-attachments/assets/19b730f5-3751-4396-a6be-9a3bf5de9f5f"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
