liupc commented on issue #23672: [SPARK-26750]Estimate memory overhead with multi-cores URL: https://github.com/apache/spark/pull/23672#issuecomment-459321568 @srowen Yes, you are right, I think if we suppose with the `MEMORY_OVERHEAD_FRACTOR`(fixed to 0.1) user app can runs well at 1 core, and user request N(core number) times of the heap memory for N times the number of cores. then everything should work well. But what if the user submit an application with N cores directly and the default fractor can not satisfy? due to lacking the basic knowlege of how much offheap size should be requested, the user might need to constantly try different config values like `spark.executor.memoryOverhead` until the application runs successfully. I was wondering if we can provide a easier config? applications which need 6G heap memory requires larger memoryOverhead than applications that only need 1G heap memory. If the fixed fractor can not satisfy, then users don't know where(0.1 * memoryOverhead) to begin increasing the config like `spark.executor.memoryOverhead`, because the MEMORY_OVERHEAD_FRACTOR is an internal fixed config, and if running with multi-cores the final config value should also multiplies the number of cores, so adjusting the config value with delta size may require more attempt. Maybe we should add a formula like 0.1 * heapMemory + biasDelta * cores ? that seems easier to change, and no matter whether the app are running with multi-cores.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
