potiuk commented on code in PR #68788:
URL: https://github.com/apache/airflow/pull/68788#discussion_r4186224067
##########
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py:
##########
@@ -349,6 +350,16 @@ def get_or_create_spark_crd(self, launcher:
CustomObjectLauncher, context) -> k8
)
return driver_pod
+ def _apply_xcom_sidecar_to_template(self, template_body: dict) -> None:
+ if self.do_xcom_push:
+ self.log.debug("Adding xcom sidecar to driver pod spec in task
%s", self.task_id)
+ driver_template = template_body["spark"]["spec"]
+ template_body["spark"]["spec"] =
add_sidecar_to_spark_operator_pod_spec(
Review Comment:
This doesn't survive templates that use the documented top-level
`kubernetes` section, as the `spark_job_template` in the docs does.
`CustomObjectLauncher.get_body()` then sets `spec["volumes"]` and the driver's
`volumeMounts` from that section (`custom_object_launcher.py:244–255`). That
replaces the XCom volume and driver mount injected here, but keeps the sidecar,
whose mount then points at a volume that no longer exists, so Kubernetes
rejects the driver pod. The tests only use templates without a `kubernetes`
section. Could you inject after `get_body()` has merged the `kubernetes`
section, or merge into its lists, and add a test with a `kubernetes` section?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
