opensky2023 commented on issue #76573:
URL: https://github.com/apache/airflow/issues/76573#issuecomment-6677902375
### Title
**[Bug] Long-running tasks (>24h) unstable on mixed Celery/K8sExecutor: 24h
worker switch (Celery) & heartbeat self-destruct on API restart (K8s)**
### Body
#### Summary
Long-running training tasks (>24h) fail reliably on Airflow 3.2.2 with mixed
`CeleryExecutor` + `KubernetesExecutor`.
- Default Celery path causes **24h worker switch** and eventual
self-destruction (~3 days).
- Explicit `KubernetesExecutor` avoids 24h issue but triggers **cascading
pod deletion** when `airflow-api-server` briefly restarts (SQLAlchemy pool
exhaustion), due to Task SDK heartbeat timeout (`Too many failed heartbeats` ->
SIGKILL).
#### Environment
- **Airflow**: 3.2.2
- **Executors**: Celery (default) + Kubernetes (explicit)
- **Broker**: Redis
- **Key Configs** (all set to 30d to mitigate timeouts):
- `jwt_expiration_time = 2592000`
- `[celery_broker_transport_options] visibility_timeout = 2592000`
- `scheduler.task_queued_timeout = 2592000`
- **Task**: Long-running (KPO), expected >24h.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
