nathadfield opened a new issue, #61130:
URL: https://github.com/apache/airflow/issues/61130
### Apache Airflow version
main (development)
### If "Other Airflow 3 version" selected, which one?
_No response_
### What happened?
When a deferrable sensor with `soft_fail=True` times out, the task **fails**
with `AirflowSensorTimeout` instead of being marked as **SKIPPED**. This is a
regression from Airflow 2.x where the same configuration correctly skipped the
task and downstream tasks.
**Actual behavior in Airflow 3.x:**
- Task state: `FAILED`
- Error: `AirflowSensorTimeout: Trigger timeout`
- Log shows: `TaskDeferralTimeout: Trigger timeout` in
`BaseSensorOperator.resume_execution()`
- Downstream tasks: Not executed
**Workaround:** Setting `deferrable=False` forces the sensor to use
traditional poke mode, which correctly respects `soft_fail=True` on timeout.
### What you think should happen instead?
A sensor configured with `soft_fail=True` should be marked as `SKIPPED` when
it times out, regardless of whether it's running in deferrable mode or
traditional poke mode. The `soft_fail` parameter should work consistently
across all sensor modes.
**Expected behavior (works in Airflow 2.x):**
- Task state: `SKIPPED`
- Downstream tasks: `SKIPPED` (according to trigger rules)
- No `AirflowSensorTimeout` exception raised
### How to reproduce
```python
from datetime import datetime, timedelta
from airflow.providers.standard.sensors.time_delta import TimeDeltaSensor
from airflow.providers.standard.operators.empty import EmptyOperator
from airflow.sdk import DAG
with DAG(
dag_id='test_sensor_soft_fail',
start_date=datetime(2024, 1, 1),
schedule=None,
) as dag:
# Sensor waiting for 1 hour but will timeout after 10 seconds
sensor = TimeDeltaSensor(
task_id='test_timedelta_sensor',
delta=timedelta(hours=1), # Wait for 1 hour
timeout=10, # But timeout after 10 seconds
soft_fail=True, # Should skip on timeout
deferrable=True, # Use deferrable mode
)
downstream = EmptyOperator(task_id='downstream_task')
sensor >> downstream
```
**Steps:**
1. Run the DAG with `deferrable=True` (as shown above)
2. Wait for sensor to timeout after 10 seconds
3. Observe task **FAILS** with `AirflowSensorTimeout` and downstream task is
not executed
4. Change `deferrable=True` to `deferrable=False` in the sensor configuration
5. Run the DAG again with `deferrable=False`
6. Wait for sensor to timeout after 10 seconds
7. Observe task is marked as **SKIPPED** and downstream task is also skipped
(correct behavior)
**Expected behavior (Airflow 2.x):** Task skips on timeout regardless of
deferrable mode
**Actual behavior (Airflow 3.x):**
- With `deferrable=True`: Task FAILS on timeout
- With `deferrable=False`: Task SKIPS on timeout (correct)
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
