GitHub user daniyarndh created a discussion: [Proposal] Self-Healing Retries
Currently, Airflow task retries are **passive and fixed**. When a task fails (e.g., `retries=3, retry_delay=5m`), Airflow re-executes the exact same code with the exact same executor configuration and resource requests. For transient code bugs, this is fine. However, in modern data engineering, many failures are caused by **transient infrastructure and operational bottlenecks**: 1. **Out-of-Memory Errors:** A batch job processes 20% more data than usual, triggering a Kubernetes `OOMKilled` signal. Retrying with the same memory allocation fails 3 times in a row, paging on-call engineers at 2 AM. 2. **API Rate Limits:** Hitting an endpoint with standard 5-minute retries often extends rate-limit penalties or results in IP bans. 3. **Database Deadlocks / Lock Timeouts:** Repeating a task instantly on the same high-concurrency worker queue prolongs resource contention. Engineers routinely over-provision resources, for instance, requesting 32 GB RAM for a task that needs 4 GB on 29 out of 30 days, solely to prevent edge-case retry failures - wasting substantial compute budget. ## Proposed Solution We propose introducing **Self-Healing Retries** capability. Instead of treating all retries identically, Airflow should allow tasks to dynamically mutate their executor specs, execution queues, or retry strategies based on the caught exception or exit code. As a result, Airflow can automatically double the memory allocation after an Out-of-Memory crash, back off during API rate limits, or move jobs to a low-concurrency queue during database locks. Airflow here acts as the **decision brain** (intercepting error classes and altering retry specs), while the underlying executor (Kubernetes, Celery, ECS) acts as the **execution muscle** (provisioning the updated container). GitHub link: https://github.com/apache/airflow/discussions/73638 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
