GitHub user Sanjay-EPG created a discussion: Conditional Retry Mechanism or Fail Fast on Retry
I was looking for a way to implement a conditional retry mechanism in Airflow, where a task retries only under certain failure conditions (e.g. transient network errors or state mismatch) but stops immediately on others (e.g. data validation errors or invalid API keys or any other valid errors). Airflow currently only supports static retry configuration via `retries `and `retry_delay` on the operator. There is no built-in way to inspect the failure reason and decide whether a retry should proceed. - Though we could have `on_retry_callback `to decide whether the error is retriable, it is not possible to stop retries Core problem: raising an exception inside a task does not stop retries, Airflow will retry regardless of the exception type, up to the configured retries count. GitHub link: https://github.com/apache/airflow/discussions/66924 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
