Re: spark.executor.pyspark.memory not added to the executor resource request on Kubernetes

... using spark 3.2.1 From: Shay Elbaz Sent: Tuesday, July 19, 2022 1:26 PM To: user@spark.apache.org Cc: Jeffrey O'Donoghue Subject: [EXTERNAL] spark.executor.pyspark.memory not added to the executor resource request on Kubernetes ATTENTION: This email

spark.executor.pyspark.memory not added to the executor resource request on Kubernetes

Hi, We are trying tune executor memory on Kubernetes. Specifically, 8g for the jvm, 8g for the python process, and additional 500m overhead: --conf spark.executor.memory=8g --conf spark.executor.pyspark.memory=8g --conf spark.executor.memoryOverhead=500m According the docs, the executor pods