I forgot top ask which k8s cluster are you using, assuming some clod vendor
view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 10 Mar 2023 at 18:00, Ismail Yenigul <ismailyeni...@gmail.com> wrote: > and If you look at the code > > > https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194 > > .editOrNewResources() > .addToRequests("memory", executorMemoryQuantity) > .addToLimits("memory", executorMemoryQuantity) > .addToRequests("cpu", executorCpuQuantity) > .addToLimits(executorResourceQuantities.asJava) > .endResources() > > addToRequests and addToLimits for memory have the same value. > maybe it is by design. but can I set custom values for them if I use > podtemplate? > > > > Ismail Yenigul <ismailyeni...@gmail.com>, 10 Mar 2023 Cum, 20:52 > tarihinde şunu yazdı: > >> Hi, >> using spark version v.3.1.2 >> >> spark.executor.memory is set. >> But the problem is not setting spark.executor.memory, the problem is that >> whatever value I set spark.executor.memory, >> spark executor pod has the same value for resources.limit.memory and >> resources.request.memory. >> I want to be able to set different values for them. >> >> >> >> >> Mich Talebzadeh <mich.talebza...@gmail.com>, 10 Mar 2023 Cum, 20:44 >> tarihinde şunu yazdı: >> >>> What are those currently set in spark-submit and which spark version on >>> k8s >>> >>> --conf spark.driver.memory=2000m \ >>> --conf spark.executor.memory=2000m \ >>> >>> HTH >>> >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> >>> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <ismailyeni...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> There is a cpu parameter to set spark executor on k8s >>>> spark.kubernetes.executor.limit.cores and >>>> spark.kubernetes.executor.request.cores >>>> but there is no parameter to set memory request different then limits >>>> memory (such as spark.kubernetes.executor.request.memory) >>>> For that reason, >>>> spark.executor.memory is assigned to requests.memory and limits.memory >>>> like the following >>>> >>>> Limits: >>>> memory: 5734MiRequests: >>>> cpu: 4 >>>> memory: 5734Mi >>>> >>>> >>>> Is there any special reason to not have >>>> spark.kubernetes.executor.request.memory parameter? >>>> and can I use spark.kubernetes.executor.podTemplateFile parameter to >>>> set smaller memory request than the memory limit in pod template file? >>>> >>>> >>>> Limits: >>>> memory: 5734MiRequests: >>>> cpu: 4 >>>> memory: 1024Mi >>>> >>>> >>>> Thanks >>>> >>>>
