There is a very good reason for this. It is recommended using k8s that you set memory request and limit to the same value, set a cpu request, but not a cpu limit. More info here https://home.robusta.dev/blog/kubernetes-memory-limit
BR, Martin ________________________________ From: Mich Talebzadeh <mich.talebza...@gmail.com> Sent: Friday, March 10, 2023 20:25 To: Ismail Yenigul <ismailyeni...@gmail.com> Cc: dev@spark.apache.org <dev@spark.apache.org> Subject: Re: spark executor pod has same memory value for request and limit EXTERNAL SENDER. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide your username or password. agreed. need to be enhanced! HTH [https://ci3.googleusercontent.com/mail-sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fmich-talebzadeh-ph-d-5205b2%2F&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QFeuv8QaKxfTbMhwQ78Bdwh%2Bagfb1m9CvLF%2BFNmPtLQ%3D&reserved=0> https://en.everybodywiki.com/Mich_Talebzadeh<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.everybodywiki.com%2FMich_Talebzadeh&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFJALy5l0wwTnWq8bXi59mLiH1YaRQsjlpLB%2FH04VPY%3D&reserved=0> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 10 Mar 2023 at 19:15, Ismail Yenigul <ismailyeni...@gmail.com<mailto:ismailyeni...@gmail.com>> wrote: Hi Mich, The issue is here there is no parameter to set executor pod request memory value. Currently we have only one parameter which is spark.executor.memory and it set pod resources limit and requests. Mich Talebzadeh <mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>>, 10 Mar 2023 Cum, 22:04 tarihinde şunu yazdı: Yes, both EKS and GKE (Google) are on 3.1.2 so I am not sure those parameters will work :( [https://ci3.googleusercontent.com/mail-sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fmich-talebzadeh-ph-d-5205b2%2F&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QFeuv8QaKxfTbMhwQ78Bdwh%2Bagfb1m9CvLF%2BFNmPtLQ%3D&reserved=0> https://en.everybodywiki.com/Mich_Talebzadeh<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.everybodywiki.com%2FMich_Talebzadeh&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFJALy5l0wwTnWq8bXi59mLiH1YaRQsjlpLB%2FH04VPY%3D&reserved=0> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 10 Mar 2023 at 19:01, Ismail Yenigul <ismailyeni...@gmail.com<mailto:ismailyeni...@gmail.com>> wrote: Hi Mich, it is on AWS EKS Mich Talebzadeh <mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>>, 10 Mar 2023 Cum, 21:11 tarihinde şunu yazdı: I forgot top ask which k8s cluster are you using, assuming some clod vendor [https://ci3.googleusercontent.com/mail-sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fmich-talebzadeh-ph-d-5205b2%2F&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QFeuv8QaKxfTbMhwQ78Bdwh%2Bagfb1m9CvLF%2BFNmPtLQ%3D&reserved=0> https://en.everybodywiki.com/Mich_Talebzadeh<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.everybodywiki.com%2FMich_Talebzadeh&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFJALy5l0wwTnWq8bXi59mLiH1YaRQsjlpLB%2FH04VPY%3D&reserved=0> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 10 Mar 2023 at 18:00, Ismail Yenigul <ismailyeni...@gmail.com<mailto:ismailyeni...@gmail.com>> wrote: and If you look at the code https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fblob%2Fe64262f417bf381bdc664dfd1cbcfaa5aa7221fe%2Fresource-managers%2Fkubernetes%2Fcore%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fdeploy%2Fk8s%2Ffeatures%2FBasicExecutorFeatureStep.scala%23L194&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j1uFQjXYZYArgc7JP5SN88lcs3paDvUCa7JEcl%2FyHko%3D&reserved=0> .editOrNewResources() .addToRequests("memory", executorMemoryQuantity) .addToLimits("memory", executorMemoryQuantity) .addToRequests("cpu", executorCpuQuantity) .addToLimits(executorResourceQuantities.asJava) .endResources() addToRequests and addToLimits for memory have the same value. maybe it is by design. but can I set custom values for them if I use podtemplate? Ismail Yenigul <ismailyeni...@gmail.com<mailto:ismailyeni...@gmail.com>>, 10 Mar 2023 Cum, 20:52 tarihinde şunu yazdı: Hi, using spark version v.3.1.2 spark.executor.memory is set. But the problem is not setting spark.executor.memory, the problem is that whatever value I set spark.executor.memory, spark executor pod has the same value for resources.limit.memory and resources.request.memory. I want to be able to set different values for them. Mich Talebzadeh <mich.talebza...@gmail.com<mailto:mich.talebza...@gmail.com>>, 10 Mar 2023 Cum, 20:44 tarihinde şunu yazdı: What are those currently set in spark-submit and which spark version on k8s --conf spark.driver.memory=2000m \ --conf spark.executor.memory=2000m \ HTH [https://ci3.googleusercontent.com/mail-sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fmich-talebzadeh-ph-d-5205b2%2F&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QFeuv8QaKxfTbMhwQ78Bdwh%2Bagfb1m9CvLF%2BFNmPtLQ%3D&reserved=0> https://en.everybodywiki.com/Mich_Talebzadeh<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.everybodywiki.com%2FMich_Talebzadeh&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532855035%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7btUJdT1bquZF%2F%2FOGTWuHeHCw94KA7AwUDAHZgIqPsM%3D&reserved=0> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <ismailyeni...@gmail.com<mailto:ismailyeni...@gmail.com>> wrote: Hi, There is a cpu parameter to set spark executor on k8s spark.kubernetes.executor.limit.cores and spark.kubernetes.executor.request.cores but there is no parameter to set memory request different then limits memory (such as spark.kubernetes.executor.request.memory) For that reason, spark.executor.memory is assigned to requests.memory and limits.memory like the following Limits: memory: 5734Mi Requests: cpu: 4 memory: 5734Mi Is there any special reason to not have spark.kubernetes.executor.request.memory parameter? and can I use spark.kubernetes.executor.podTemplateFile parameter to set smaller memory request than the memory limit in pod template file? Limits: memory: 5734Mi Requests: cpu: 4 memory: 1024Mi Thanks
