raphaelauv commented on PR #41047:
URL: https://github.com/apache/airflow/pull/41047#issuecomment-2269253971
`But we should assume that most people are at 2.7.1 + now`
so most people suffer from this silent bug and should not trust soft_fail (
I had to remove soft_fail to all my sensors and add a ShortCircuitOperator
checking the status of previous task and check if it's a `AirflowSensorTimeout`
) ->
```python
from airflow.operators.empty import EmptyOperator
from airflow.sensors.python import PythonSensor
from airflow.operators.python import ShortCircuitOperator
from airflow.utils.trigger_rule import TriggerRule
from pendulum import today
from airflow import DAG
def a():
# return False # task_c will run
raise ValueError("toto") # task_c will not run
def b(error: str | None):
if error == "AirflowSensorTimeout":
return True
else:
return False
def sav_error(context):
exception_name = context["exception"].__class__.__name__
context.get('task_instance').xcom_push("error", exception_name)
with DAG(
dag_id='example',
schedule_interval='0 0 * * *',
start_date=today("UTC").add(days=-5)):
a = PythonSensor(task_id="a", python_callable=a,
on_failure_callback=sav_error, timeout=5, poke_interval=5)
b = ShortCircuitOperator(task_id="b", python_callable=b,
trigger_rule=TriggerRule.ALL_DONE, op_kwargs={
'error': "{{ ti.xcom_pull(task_ids='a', key='error') }}"})
c = EmptyOperator(task_id="c")
a >> b >> c
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
