[
https://issues.apache.org/jira/browse/SPARK-51020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ali Sabir updated SPARK-51020:
------------------------------
Description:
We are attempting to use a pod template for the executor to expose a JMX port
for exporting metrics. While the pod template works for the driver — with the
JMX port being created and the template successfully mounted at
/opt/spark/pod-template — the same template is not being applied to the
executor.
Both the driver and executor pod templates are provided in the attached files.
The following Spark properties are being passed during job submission:
spark.kubernetes.driver.podTemplateFile=s3a://mybucket/code/driver-pod-template.yaml
spark.kubernetes.executor.podTemplateFile=s3a://mybucket/code/exec-pod-template.yaml
Despite these configurations, the executor pod template does not appear to take
effect.
I have attached the details of the executor and driver pods as they appear when
launched. Additionally, I am including the Spark configuration passed to the
driver during job submission.
was:
We are attempting to use a pod template for the executor to expose a JMX port
for exporting metrics. While the pod template works for the driver — with the
JMX port being created and the template successfully mounted at
/opt/spark/pod-template — the same template is not being applied to the
executor.
Both the driver and executor pod templates are provided in the attached files.
The following Spark properties are being passed during job submission:
spark.kubernetes.driver.podTemplateFile=s3a://mybucket/code/driver-pod-template.yaml
spark.kubernetes.executor.podTemplateFile=s3a://mybucket/code/exec-pod-template.yaml
Despite these configurations, the executor pod template does not appear to take
effect.
> User-specified executor pod templates are not working in Spark 3.3.1.
> ---------------------------------------------------------------------
>
> Key: SPARK-51020
> URL: https://issues.apache.org/jira/browse/SPARK-51020
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Spark Core
> Affects Versions: 3.3.1
> Reporter: Ali Sabir
> Priority: Minor
> Attachments: driver-pod-template.yml, exec-pod-template.yml
>
>
> We are attempting to use a pod template for the executor to expose a JMX port
> for exporting metrics. While the pod template works for the driver — with the
> JMX port being created and the template successfully mounted at
> /opt/spark/pod-template — the same template is not being applied to the
> executor.
> Both the driver and executor pod templates are provided in the attached
> files. The following Spark properties are being passed during job submission:
> spark.kubernetes.driver.podTemplateFile=s3a://mybucket/code/driver-pod-template.yaml
> spark.kubernetes.executor.podTemplateFile=s3a://mybucket/code/exec-pod-template.yaml
> Despite these configurations, the executor pod template does not appear to
> take effect.
>
> I have attached the details of the executor and driver pods as they appear
> when launched. Additionally, I am including the Spark configuration passed to
> the driver during job submission.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
