khmelevskiy opened a new issue, #73164:
URL: https://github.com/apache/airflow/issues/73164
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
When a `KubernetesExecutor` worker pod is OOM-killed, the executor collects
the failure details correctly. The scheduler log shows:
```
Task dc_cls__...process_area_WE_Golden_Ocean.5 failed in pod
airflow-data-cube-cls/dc-cls-...-bvma68of.
Pod phase: Failed, reason: None, message: None, container_type: main,
container_name: base,
container_state: terminated, container_reason: OOMKilled, container_message:
None, exit_code: 137
```
But the `termination_reason` that is passed to the scheduler via
`event_buffer` (and ends up in the task's Event log as `Extra info`) is built
only from `pod.status.reason`:
```python
# kubernetes_executor.py, _change_state()
termination_reason = f"Pod failed because of {pod_reason}"
```
Kubernetes leaves `pod.status.reason` empty for container-level failures
such as OOMKilled or a non-zero exit, so the Event log row (`event="state
mismatch"`) reads:
```
... Learn more:
https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally
Extra info: Pod failed because of None
```
The task log is empty too (the process was SIGKILLed), so from the UI / REST
API an OOM kill is indistinguishable from any other external kill. The only
place the real reason exists is the scheduler log.
Introduced by #60800, whose stated goal was exactly to surface OOM / Evicted
in the Event log. The same line is still in `main` and in
`apache-airflow-providers-cncf-kubernetes==10.22.0`.
**Reproduce:** run any task under `KubernetesExecutor` with a memory limit
lower than the task needs (e.g. `executor_config` with `limits.memory: 64Mi`
and a task that allocates 200 MiB), wait for the failure, open the task's Event
log tab.
### What you think should happen instead?
Fall back to the container-level reason when the pod-level one is empty, and
include the container name / exit code, e.g.
```
Extra info: Pod failed because of OOMKilled (container: base, exit code: 137)
```
### Operating System
Debian 12 (official image base), Kubernetes v1.35.5
### Deployment
Other 3rd-party Helm chart
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==10.21.1 (same code in 10.22.0 and
`main`)
### Kubernetes Version
v1.35.5
### Anything else?
Related: #66405 (AIP-97, typed failure causes) is the broader effort; this
issue is only about the string that already exists being built from the wrong
field.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
