bnspopi opened a new pull request, #77309: URL: https://github.com/apache/airflow/pull/77309
The [Timeouts section of the task docs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#timeouts) says that retrying a sensor does not reset its `timeout`. In Airflow 3 it does: every try is its own task instance row with its own id, and the reschedules of a try are looked up by that id, so a retry sees none of the reschedules of the tries before it. A sensor in `reschedule` mode with `retries=2` and `timeout=12h` can wait about 36 hours. That is the reschedule-mode half of #77255. The Task SDK already meant to anchor the timeout to the first try. `RuntimeTaskInstance.get_first_reschedule_date` computes `first_try_number = max_tries - retries + 1` and sends it as `try_number` with its request for the first reschedule date. Two things stopped that from working: - The server ignored it. `GET /task-reschedules/{task_instance_id}/start_date` has no `try_number` parameter and filters on the task instance id only. - On a retry the request was never made. `task_reschedule_count` is 0 for a new try, so the method returned `None` before it got that far. This change: - **Server**: the endpoint takes an optional `try_number`. With it, the reschedules of earlier tries of the same task (same `dag_id`, `task_id`, `run_id`, `region_id`, `region_index`) count as well, from that try up to the current one. Without it the result is the same as before. - **Task SDK**: on a retry, `get_first_reschedule_date` asks the supervisor for the date since the first try. On the first try it behaves as before, so there is no extra request there. Clearing a task still starts a new window: clearing sets `max_tries` to the last try plus `retries`, which moves `first_try_number` past the tries before the clear. The reschedule limits of a retry policy (`max_reschedules`, `reschedule_timeout`) are documented as counting per try, so `_reschedule_limit_reached` now calls a helper that keeps looking at the current try only. Not covered: - Deferrable sensors. They pass `timeout=timedelta(seconds=self.timeout)` to `self.defer()` themselves, measured from the moment each try defers. That is the other half of #77255 and needs changes in the providers, so this PR does not close the issue. - Sensors in `poke` mode, which measure their timeout inside one process. There is one more request to the supervisor per sensor execution on a retry. I did not add an API version change for the new query parameter: clients already send it, and older API versions ignored it only by accident. Tell me if you would rather have one. Tests, run on Linux with SQLite: - `TestGetRescheduleStartDate`: 5 new cases, 8 passed. Without the change, the two cases that reach back to an earlier try fail. - `test_task_runner.py -k first_reschedule_date`: 4 new cases, 7 passed. Without the change, 3 fail. - All of `test_task_instances.py` in the execution API: 365 passed, 7 skipped. - `test_task_runner.py` and `bases/test_sensor.py`: 634 passed, 30 failed. The 30 are the `test_xcom_pull[dataframe_value-...]` cases, which fail the same way without the change in my environment. related: #77255 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Claude Code) Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
