ThePresence opened a new pull request, #49505:
URL: https://github.com/apache/airflow/pull/49505
feat(kubernetes): inject XCom sidecar in SparkKubernetesOperator when
do_xcom_push=True
## What?
Adds XCom support for SparkKubernetesOperator by injecting an `emptyDir`
volume, mounting `/airflow/xcom`, and spawning the standard
`airflow-xcom-sidecar` container when `do_xcom_push=True`.
## Why?
Fixes issue #39184 where SparkKubernetesOperator tasks hang indefinitely
because no sidecar is injected to read `/airflow/xcom/return.json`.
## How?
- In `execute()`, when `do_xcom_push=True`, append:
- A volume named `xcom` with `emptyDir`.
- A `volumeMount` for the driver spec at `/airflow/xcom`.
- A sidecar container `airflow-xcom-sidecar` using the Alpine image and
minimalist sleep loop.
- Calls `super().execute()` to trigger pod launch and sidecar monitoring.
## Testing
- Added
`tests/unit/providers/cncf/kubernetes/test_spark_kubernetes_operator_xcom.py`
to assert the presence of the XCom volume, mount, and sidecar in the rendered
SparkApplication spec.
## Backwards‑Incompatible?
- Added newsfragment `39184.significant.rst` to document new behavior for
operator authors.
Closes #39184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
