vinay-kumar-uppala opened a new issue, #71369:
URL: https://github.com/apache/airflow/issues/71369
### Under which category would you file this issue?
Helm chart
### Apache Airflow version
3.2.2
### What happened and how to reproduce it?
When using `KubernetesPodOperator` with `do_xcom_push=True`, the XCom sidecar
container starts correctly and the XCom value is **read successfully** from
`/airflow/xcom/return.json` (confirmed via `cat` in the task logs). However,
immediately after the read, Airflow's `extract_xcom_kill` step attempts to
terminate the sidecar's idle loop process via `kill -2` over `kubectl exec`,
and this consistently fails with `Permission denied`:
```
INFO - The xcom sidecar container has started.
INFO - Running command... if [ -s /airflow/xcom/return.json ]; then cat
/airflow/xcom/return.json; else echo __airflow_xcom_result_empty__; fi
INFO - Running command... kill -2 $(pgrep -u $(id -u) -f 'sh')
WARNING - stderr from command: sh: can't kill pid 314: Permission denied
INFO - Primary kill command failed, trying fallback command
INFO - Running command... for f in /proc/[0-9]*/comm; do [ -O $f ] && read c
< $f && [ "$c" = "sh" ] && pid=${f%/comm} && kill -2 ${pid##*/}; done
WARNING - stderr from command: sh: can't kill pid 314: Permission denied
INFO - Deleting pod: push-xcom-pod-6jcmp6lx
ERROR - Task failed with exception
```
Both the primary kill command and the documented fallback command fail
identically. Because the exception is raised inside `extract_xcom()`
**after** the value has already been read from the file but **before** it is
returned to the caller, the task is marked as `Failed` even though the XCom
payload was technically retrieved from the pod. As a result:
- The XCom value is never persisted to the metadata DB.
- The task instance shows `Failed`.
- Downstream tasks depending on this task are marked `Upstream Failed` and
never run.
Both the primary kill command and the documented fallback command fail
identically. Because the exception is raised inside `extract_xcom()`
**after** the value has already been read from the file but **before** it is
returned to the caller, the task is marked as `Failed` even though the XCom
payload was technically retrieved from the pod. As a result:
- The XCom value is never persisted to the metadata DB.
- The task instance shows `Failed`.
- Downstream tasks depending on this task are marked `Upstream Failed` and
never run.
Full traceback:
```
AirflowException: Pod push-xcom-pod-6jcmp6lx returned a failure.
...
File ".../airflow/providers/cncf/kubernetes/operators/pod.py", line 770, in
execute_sync
File ".../airflow/providers/cncf/kubernetes/operators/pod.py", line 1151, in
post_complete_action
File ".../airflow/providers/cncf/kubernetes/operators/pod.py", line 1224, in
cleanup
PodCommandException: Command failed with stderr: sh: can't kill pid 314:
Permission denied
File ".../airflow/providers/cncf/kubernetes/operators/pod.py", line 763, in
execute_sync
File ".../airflow/providers/cncf/kubernetes/operators/pod.py", line 677, in
extract_xcom
File ".../airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 974,
in extract_xcom
File ".../tenacity/__init__.py", line 331, in wrapped_f
File ".../tenacity/__init__.py", line 470, in __call__
File ".../tenacity/__init__.py", line 371, in iter
File ".../tenacity/__init__.py", line 393, in <lambda>
File ".../concurrent/futures/_base.py", line 449, in result
File ".../concurrent/futures/_base.py", line 401, in __get_result
File ".../tenacity/__init__.py", line 473, in __call__
File ".../airflow/providers/cncf/kubernetes/utils/pod_manager.py", line
1046, in extract_xcom_kill
File ".../airflow/providers/cncf/kubernetes/utils/pod_manager.py", line
1063, in _exec_pod_command
PodCommandException: Command failed with stderr: sh: can't kill pid 314:
Permission denied
File ".../airflow/providers/cncf/kubernetes/utils/pod_manager.py", line
1043, in extract_xcom_kill
File ".../airflow/providers/cncf/kubernetes/utils/pod_manager.py", line
1063, in _exec_pod_command
```
Minimal DAG (see attached file, `xcom_kpo_repro_dag.py`):
```python
from __future__ import annotations
from datetime import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.pod import
KubernetesPodOperator
with DAG(
dag_id="xcom_kpo_repro",
schedule=None,
catchup=False,
) as dag:
push_xcom_task = KubernetesPodOperator(
task_id="push_xcom_task",
name="push-xcom-pod",
namespace="<namespace>",
image="python:3.11-slim",
image_pull_policy="IfNotPresent",
cmds=["python", "-c"],
arguments=[
"import json; "
"data = {'message': 'hello from the pod', 'count': 42}; "
"f = open('/airflow/xcom/return.json', 'w'); "
"json.dump(data, f); "
"f.close()"
],
do_xcom_push=True,
get_logs=True,
on_finish_action="delete_pod",
in_cluster=True,
)
```
Steps:
1. Deploy the DAG above to a cluster running Airflow 3.2.2 with the
`apache-airflow-providers-cncf-kubernetes` provider, on a k3s cluster
(containerd runtime).
2. Trigger the DAG.
3. Observe the task logs: the XCom value is read successfully via `cat`
inside the sidecar, but the subsequent `kill -2` cleanup step fails with
`Permission denied`, and the task is marked `Failed`.
To confirm this is unrelated to XCom content or main-container behavior, we
also reproduced it with the `base` container modified to `sleep()` after
writing the XCom file (to keep the pod alive), then manually ran:
```bash
kubectl exec -n <namespace> <pod-name> -c airflow-xcom-sidecar -- sh -c "id;
ps aux; kill -2 1"
```
against the live sidecar container — same `Permission denied` result outside
of Airflow's own retry/exec logic, confirming it is not specific to
Airflow's particular invocation of the kill command.
### What you think should happen instead?
Either:
1. The kill step should succeed against the sidecar's own idle process
(`trap "exit 0" INT; while true; do sleep 1; done;`), since it is the
same container's own PID 1 and no custom `securityContext` is set on the
pod (see pod spec below — `security_context` is `None` at both the pod
and container level, as submitted by the operator itself), **or**
2. If a signal-based teardown is inherently unreliable across container
runtimes/environments (e.g. due to runtime-level signal isolation between
an exec session and PID 1), the sidecar teardown should fall back to a
non-signal-based mechanism (e.g. deleting the pod directly) without
raising an exception that fails the *task*, since the XCom value has
already been successfully extracted at that point in the code path.
At minimum, a failure in the cleanup/kill step after a successful XCom read
should not discard the already-retrieved XCom value or fail the task.
### Operating System
linux
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
- **Airflow version:** 3.2.2
- **Kubernetes distribution:** k3s
- **Container runtime:** containerd (default for k3s)
- **Namespace Pod Security Admission label:** none set (confirmed via
`kubectl get ns <namespace> -o yaml` — no
`pod-security.kubernetes.io/enforce` label present)
- **Pod-level / container-level `securityContext`:** `None` in both the
operator-submitted spec and the live pod spec (no `runAsUser`,
`runAsNonRoot`, `capabilities`, or `seccompProfile` set anywhere in the
submitted spec)
- **Sidecar image:** `alpine:3.23.4` (as injected automatically by the
operator)
- **`do_xcom_push`:** `True`
- **`in_cluster`:** `True`
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
