Hi everyone, I want to start a discussion about how Airflow handles task retries. Right now, our retry system is a bit too rigid for modern cloud setups. The Problem When a task fails, Airflow just tries again using the exact same settings and resources. This is fine for temporary code bugs, but it doesn't work well for infrastructure problems:
- Running out of memory: A task gets a bit more data than usual, crashes because it ran out of RAM, and fails all 3 retries. An engineer gets woken up at 2 AM for something that could be automated. - API Rate Limits: Retrying too quickly can get your IP address blocked or make the API penalty last longer. - Database Overload: Retrying a heavy query immediately on a crowded database just makes the traffic jam worse. Because of this, engineers over-provision resources. They ask for 32GB of RAM for a task that only needs 4GB most of the time, just to stop it from failing on a bad day. This wastes a lot of money and cloud compute. Proposed Solution: "Self-Healing" Retries I propose letting Airflow change task settings dynamically between retries based on the specific error or exit code. Think of it like this: - Airflow is the brain: It sees *why* the task failed and updates the settings for the next attempt. - The Executor is the muscle: It spins up the new task with these updated settings. Exam*ples:* - If a task dies from running out of memory, Airflow automatically doubles the RAM limit for the next retry. - If a task hits a rate limit, Airflow switches to a slower, exponential backoff schedule. - If a database is jammed, Airflow moves the retry to a slower, low-priority queue. I know this changes how Airflow schedules things, so it will likely need a formal Airflow Improvement Proposal (AIP). I would love to get your initial thoughts here on the mailing list. I am also planning to join today's dev call (Sep 24) to say hello, but not sure if I will get a chance to introduce the concept, and answer any quick questions. Best regards, Daniyar
