What are those currently set in spark-submit and which spark version on k8s
--conf spark.driver.memory=2000m \
--conf spark.executor.memory=2000m \
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <ismailyeni...@gmail.com>
wrote:
> Hi,
>
> There is a cpu parameter to set spark executor on k8s
> spark.kubernetes.executor.limit.cores and
> spark.kubernetes.executor.request.cores
> but there is no parameter to set memory request different then limits
> memory (such as spark.kubernetes.executor.request.memory)
> For that reason,
> spark.executor.memory is assigned to requests.memory and limits.memory
> like the following
>
> Limits:
> memory: 5734MiRequests:
> cpu: 4
> memory: 5734Mi
>
>
> Is there any special reason to not have
> spark.kubernetes.executor.request.memory parameter?
> and can I use spark.kubernetes.executor.podTemplateFile parameter to set
> smaller memory request than the memory limit in pod template file?
>
>
> Limits:
> memory: 5734MiRequests:
> cpu: 4
> memory: 1024Mi
>
>
> Thanks
>
>
