eddiemaru-101 commented on issue #54479:
URL: https://github.com/apache/airflow/issues/54479#issuecomment-3224173936
# Root Cause Analysis – Airflow 3.0 CeleryExecutor Issue in K8s
---
## Overview
After analyzing the logs and code, the root cause of this issue appears not
to be related to `visibility_timeout`, but rather to logging configuration
changes introduced in Airflow 3.0.
## Key Findings
### 1. Airflow 3.0 Logging Changes
**File:**
`providers/celery/src/airflow/providers/celery/executors/default_celery.py`
(lines 97-99)
```python
# In order to not change anything pre Task Execution API, we leave this
setting as it was (unset) in Airflow2
if AIRFLOW_V_3_0_PLUS:
DEFAULT_CELERY_CONFIG.setdefault("worker_redirect_stdouts", False)
DEFAULT_CELERY_CONFIG.setdefault("worker_hijack_root_logger", False)
```
- These settings did not exist in Airflow 2.x
- Introduced in 3.0+ for Task SDK integration
### 2. The Real Problem
- **Airflow 2.x:** Celery managed all logging (default `True`)
- **Airflow 3.0+:** Changed to `False` for Task SDK compatibility
- **K8s Environment:** This change may cause worker instability for
long-running tasks
### 3. Evidence from Logs
```
Process exited [supervisor] exit_code=-9 pid=1877111 signal_sent=SIGKILL
```
- SIGKILL indicates external termination (OS/K8s level)
- Not caused by Celery or Airflow internal timeout
- Likely related to logging handler conflicts in K8s environment
### 4. Core Conclusion
1. Worker fetches a message and starts executing the task
2. Long-running task + logging configuration (`False/False`) → stdout/stderr
→ IO buffer fills → worker blocks
3. K8s sends SIGKILL → Worker1 dies
4. `visibility_timeout` (24h) → message becomes visible again in the queue
5. Another worker can fetch the message and re-execute the task
**Summary:**
- Root cause of worker death: **stdout/stderr logging → K8s IO buffer
saturation**
- `visibility_timeout` only controls **how long the message remains
invisible before another worker can pick it up**
- Logging settings (`False/False`) and `visibility_timeout` do **not
directly conflict**
## Why It Worked in Airflow 2.10
- Celery fully controlled logging (`worker_hijack_root_logger=True` by
default)
- No Task SDK integration
- Stable logging behavior in K8s pods
## Why It Fails in Airflow 3.0.2
- Logging architecture changed for Task SDK compatibility
- K8s log collection may conflict with new logging behavior
- Long-running tasks (e.g., 36-hour tasks) expose instability
## Temporary Workarounds to Test
**Option 1: Override Celery Config**
```
# In airflow.cfg or environment variables
[celery]
extra_celery_config = {
"worker_hijack_root_logger": true,
"worker_redirect_stdouts": true
}
```
**Option 2: Task Time Limits**
```
[celery]
extra_celery_config = {
"task_soft_time_limit": 129600,
"task_time_limit": 133200,
"worker_max_tasks_per_child": 10
}
```
## Questions for Community
1. Has anyone experienced long-running task failures after upgrading to
Airflow 3.0+ with CeleryExecutor in K8s?
2. Should logging configuration be environment-aware (e.g., detect K8s and
use legacy logging for stability)?
3. Is there a better way to maintain Task SDK compatibility while ensuring
stability for long-running tasks in containerized environments?
## Proposed Solutions
Instead of only adjusting `visibility_timeout`, a code change to detect
execution environment and adjust logging settings may be needed:
- Detect K8s environment
- Apply appropriate Celery logging configuration
- Maintain Task SDK compatibility where possible
- Provide configuration override options
-"Please provide any helpful feedback.”
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
