LucaCanali commented on pull request #29477: URL: https://github.com/apache/spark/pull/29477#issuecomment-702604124
@dongjoon-hyun I see that this is a minor bug. I'd say that if there is no time or necessity to fix this during the current phase of Spark on K8s GA preparation, another option could be just to add a note in the documentation, with somehting like "Executors on K8S will not allocate extra memory for off-heap allocations based on the configuration `spark.memory.offHeap.size`, differently from what happens with executors on YARN. Users will have to allocate the extra memory needed for off-heap allocations on K8S using the configuration parameter `spark.executorMemoryOverhead` and/or `spark.kubernetes.memoryOverheadFactor`." ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
