Ferdinanddb opened a new issue, #77255: URL: https://github.com/apache/airflow/issues/77255
### Under which category would you file this issue? Task SDK ### Apache Airflow version 3.4.0 (task-sdk 1.4.0, apache-airflow-providers-standard 1.21.0) ### What happened and how to reproduce it? The [Timeouts section of the task docs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#timeouts) says that a sensor that fails for another reason (a network outage, say) "can retry up to 2 times as defined by `retries`. Retrying does not reset the `timeout`. It will still have up to 3600 seconds in total for it to succeed." In Airflow 3 every retry gets a brand new `timeout`, both for sensors in `reschedule` mode and for deferrable sensors. Both repros below use `timeout=120`, crash on try 1 about 60 s in, and retry once. **Reschedule mode** (`PythonSensor`): try 2 starts with no reschedule history and waits another full 120 s, so the sensor times out 193 s after its first poke: ``` try=1 reschedules_seen=0 now=18:04:18 try=1 reschedules_seen=1 now=18:04:48 try=1 reschedules_seen=2 now=18:05:19 -> RuntimeError: simulated crash, up_for_retry try=2 reschedules_seen=0 now=18:05:29 <- the try starts with no reschedule history try=2 reschedules_seen=1 now=18:05:59 try=2 reschedules_seen=2 now=18:06:30 try=2 reschedules_seen=3 now=18:07:00 try=2 reschedules_seen=4 now=18:07:31 AirflowSensorTimeout: Sensor has timed out; run duration of 121.99157 seconds exceeds the specified timeout of 120.0. ``` **Deferrable mode** (`ExternalTaskSensor`, `deferrable=True`): try 2 defers with a deadline of its own defer time + 120 s. From the metadata DB (UTC): ``` try_number=1 start_date=17:57:59 -> deferred, RuntimeError: simulated crash at 17:59:02, up_for_retry try_number=2 start_date=17:59:12 trigger_timeout=18:01:12 <- 120 s after try 2 deferred, not after try 1 started ``` --- ### To reproduce: **Reschedule mode:** ```python from datetime import datetime, timedelta from airflow.providers.standard.sensors.python import PythonSensor from airflow.sdk import DAG def _never_ready(ti, task_reschedule_count, **_): # Try 1 crashes on its third poke (~60 s in), like a worker restart would. print(f"try={ti.try_number} reschedules_seen={task_reschedule_count} now={datetime.now():%H:%M:%S}") if ti.try_number == 1 and task_reschedule_count >= 2: raise RuntimeError("simulated crash") return False with DAG("sensor_timeout_reset_on_retry", start_date=datetime(2026, 1, 1), schedule=None, catchup=False): PythonSensor( task_id="wait", python_callable=_never_ready, mode="reschedule", poke_interval=30, timeout=120, # docs: "Retrying does not reset the timeout" retries=1, retry_delay=timedelta(seconds=10), ) ``` Run `airflow dags test sensor_timeout_reset_on_retry` (a fresh SQLite metadata DB is enough) and compare the first `try=1` line with the `AirflowSensorTimeout` time. **Deferrable mode (a real `ExternalTaskSensor`, except that try 1 crashes after ~60 s of deferral):** ```python from datetime import datetime, timedelta from airflow.providers.standard.sensors.external_task import ExternalTaskSensor from airflow.providers.standard.triggers.temporal import TimeDeltaTrigger from airflow.sdk import DAG class CrashOnceExternalTaskSensor(ExternalTaskSensor): """A real ExternalTaskSensor, except try 1 crashes after ~60 s of deferral.""" def execute(self, context): if context["ti"].try_number == 1: self.defer(trigger=TimeDeltaTrigger(timedelta(seconds=60)), method_name="crash") super().execute(context) def crash(self, context, event=None): raise RuntimeError("simulated crash") with DAG("upstream_never_runs", start_date=datetime(2026, 1, 1), schedule=None, catchup=False): pass with DAG("deferred_sensor_timeout_reset_on_retry", start_date=datetime(2026, 1, 1), schedule=None, catchup=False): CrashOnceExternalTaskSensor( task_id="wait", external_dag_id="upstream_never_runs", external_task_id=None, deferrable=True, poke_interval=30, timeout=120, # docs: "Retrying does not reset the timeout" retries=1, retry_delay=timedelta(seconds=10), ) ``` `airflow dags test` runs triggers inline and does not enforce trigger timeouts, so once try 2 has deferred, read its deadline from the metadata DB: ```sql SELECT try_number, start_date, trigger_timeout FROM task_instance WHERE dag_id = 'deferred_sensor_timeout_reset_on_retry'; ``` ### What you think should happen instead? As documented: a retry only gets what is left of `timeout`, counted from the first try. In the repros, try 2 should time out around 18:06:18 (reschedule) and its `trigger_timeout` should be about 17:59:59 (deferrable). **Reschedule mode.** The task SDK still intends this: `RuntimeTaskInstance.get_first_reschedule_date` computes `first_try_number = max_tries - retries + 1` to anchor the timeout to the first try. That anchoring can no longer work: - Each retry is a new task instance row with a new id: `TaskInstance.prepare_db_for_next_try` inserts the successor with `id=uuid7()` (`airflow-core/src/airflow/models/taskinstance.py`). - The execution API's `ti_run` counts reschedules and their first `start_date` with `.where(TaskReschedule.ti_id == task_instance_id)` (`airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py`), i.e. for the current try only. - So on a new try `task_reschedule_count` is 0, `get_first_reschedule_date` returns `None` before it reaches the `first_try_number` lookup, and `BaseSensorOperator.execute` starts the clock again. **Deferrable mode.** Deferrable sensors pass their timeout to `self.defer(timeout=timedelta(seconds=self.timeout), ...)` (e.g. `ExternalTaskSensor.execute` in the standard provider), which is measured from the moment each try defers. Nothing subtracts the time already spent in earlier tries. ### Operating System Linux ### Deployment Other ### Apache Airflow Provider(s) standard ### Versions of Apache Airflow Providers apache-airflow-providers-standard 1.21.0 ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? - Why it matters: `timeout` is the only bound on how long a sensor can wait. If each retry gets a fresh one, a sensor with `retries=2` and `timeout=12h` can wait up to ~36 h, and a DAG with a cutoff has to set `retries=0` on its sensors, giving up protection against worker crashes. - A possible fix: record when the first try started waiting (for example the first reschedule `start_date` across all tries of the same `(dag_id, run_id, task_id, map_index)` since the last clear, or a start time carried over to the successor in `prepare_db_for_next_try`), and use it in both paths: `BaseSensorOperator.execute` for reschedule/poke mode, and the timeout passed to `defer()` for deferrable sensors. Clearing a task should still start a new window, as it does today. - If the new behaviour is intended instead, the Timeouts section of the docs needs updating. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
