GitHub user mg1986jp closed the discussion with a comment: Conditional Retry
Mechanism or Fail Fast on Retry
Airflow does not have a built-in conditional retry mechanism, but you can
implement "fail fast on specific exception types" cleanly using a custom
exception and the `on_retry_callback`.
## Approach: raise `AirflowFailException` to skip remaining retries
Airflow has a built-in exception class `AirflowFailException` that marks a task
as `failed` immediately, bypassing any remaining retries. This is the idiomatic
way to distinguish "don't retry this" from "retry this".
```python
from airflow.exceptions import AirflowFailException, AirflowException
def my_operator_execute(context):
try:
result = call_external_api()
except InvalidApiKeyError as e:
# Don't retry — permanent failure
raise AirflowFailException(f"Invalid API key: {e}")
except DataValidationError as e:
# Don't retry — data problem
raise AirflowFailException(f"Validation failed: {e}")
except TransientNetworkError as e:
# Do retry — raise a normal exception
raise AirflowException(f"Network error, will retry: {e}")
```
When `AirflowFailException` is raised, Airflow sets the task state to `failed`
immediately and does **not** consume a retry slot.
## If you are wrapping a third-party operator
You can use `on_failure_callback` on the task, but since callbacks cannot
change the retry behavior retroactively, the cleanest approach is a thin
wrapper operator that catches exceptions and re-raises as
`AirflowFailException` where appropriate:
```python
from airflow.models import BaseOperator
from airflow.exceptions import AirflowFailException
class ConditionalRetryOperator(BaseOperator):
NON_RETRIABLE = (ValueError, KeyError, PermissionError)
def execute(self, context):
try:
return self._do_work(context)
except self.NON_RETRIABLE as e:
raise AirflowFailException(str(e)) from e
def _do_work(self, context):
# your actual logic here
pass
```
## What `on_retry_callback` cannot do
You are correct that `on_retry_callback` runs after Airflow has already decided
to retry — it cannot cancel the retry. It is useful for notifications or
cleanup, but not for controlling retry logic.
## Summary
| Scenario | What to raise |
|---|---|
| Transient error, should retry | `AirflowException` (or any non-fail
exception) |
| Permanent error, skip retries | `AirflowFailException` |
| Task should be marked as skipped | `AirflowSkipException` |
`AirflowFailException` is documented in the [Airflow exceptions
module](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/exceptions/index.html)
and is the intended mechanism for exactly your use case.
GitHub link:
https://github.com/apache/airflow/discussions/66924#discussioncomment-17058213
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
