LucaCanali opened a new pull request #29477: URL: https://github.com/apache/spark/pull/29477
### What changes were proposed in this pull request? Off-heap memory allocations are configured using `spark.memory.offHeap.enabled=true` and `conf spark.memory.offHeap.size=<size>`. Spark on YARN adds the off-heap memory size to the executor container resources. Spark on Kubernetes does not request the allocation of the off-heap memory. This PR proposes to make Spark on Kubernetes behave as Spark on YARN, that is by adding spark.memory.offHeap.size to the memory request for executor containers. ### Why are the changes needed? Currently, this issue can be worked around by using `spark.executor.memoryOverhead` to reserve memory for off-heap allocations. This PR proposes to provide a more clean solution and align with Spark on YARN. ### Does this PR introduce _any_ user-facing change? Yes: when running on K8S, the memory specifies in `spark.memory.offHeap.size` will be added to the executor cointainer memory request. ### How was this patch tested? A unit test is provided. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
