stegololz opened a new pull request, #75774:
URL: https://github.com/apache/airflow/pull/75774
# Fix MSGraphSensor timeout being reset on every re-defer
## What
`MSGraphSensor` could never time out. It now raises `AirflowSensorTimeout`
once `timeout`
seconds have passed since the first `execute()` of the try, so `soft_fail` /
`never_fail`
work again. It also stops passing the sensor timeout to `MSGraphTrigger`,
which used it as
the HTTP request timeout.
## Why
A defer timeout is stored as `utcnow() + timeout` (AF3 execution API
`ti_update_state`,
AF2 `TaskInstance._defer_task`), and `timeout=None` clears any earlier
deadline. The sensor
alternated between an `MSGraphTrigger` defer without a timeout (clearing the
deadline) and a
`TimeDeltaTrigger` sleep with the full timeout (restarting it), so the
deadline moved forward
on every poll and was never reached.
Separately, `timeout=self.timeout` was passed to `MSGraphTrigger`, where it
became the httpx
per-request timeout (7 days by default).
## How
- The first `execute()` records its start time and carries it in
`next_kwargs`
(`sensor_started_at`) through `execute_complete` and `retry_execute`; if
it is missing
(TIs deferred before upgrading), it falls back to now.
- Each request defer gets the remaining time as its defer timeout, so a hung
Graph call is
bounded as well.
- The deadline is checked after a not-done poll, in the same order as
`BaseSensorOperator.execute`. Past the deadline, `AirflowSensorTimeout` is
raised from the
resumed method, so core `resume_execution` still applies `soft_fail` (and
`never_fail` on
3.1+). It is not retried.
- The sleep between polls is capped at the remaining time and is always
followed by one last
poll, which is given at least `retry_delay` so that small clock
differences between workers
cannot cut it short. The sensor can therefore overrun its deadline by at
most `retry_delay`.
A done response that arrives after the deadline still succeeds.
- The timeout is counted per try, like AF3 core sensors.
- `retry_execute` now defers directly instead of calling `execute()`.
The HTTP timeout stays at the trigger/hook default (`None`); each request is
still bounded by
the remaining sensor time. A dedicated HTTP-timeout parameter can follow if
needed.
## How tested
- New time_machine unit tests: first defer timeout and start time (trigger
HTTP timeout is
`None`), deadline not moving across re-defers, sleep capped at the
remaining time,
`AirflowSensorTimeout` past the deadline, one last poll (given at least
`retry_delay`) when
resuming near, at or past the deadline, done response after the deadline,
`resume_execution` with no flag / `soft_fail` / `never_fail`,
missing-start-time fallback,
`timeout=None`.
- Existing end-to-end deferrable tests unchanged.
- Whole azure sensors suite + msgraph trigger tests: 80 passed. With the
source change
reverted, the new sensor tests fail.
- `prek run --files <changed files>` and mypy-providers pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
