+1 - it's always been a little "cargo cult" on using Airflow Exception. There was no benefits of using it - I do recall (very, very early days of mine in Airflow where I had completely no idea what I was doing) - in 1.10 there was a slight different handling in case there was "Airflow Exception" thrown and "other exceptions" - but I even can't remember what it was and it is a long gone, yet "cargo cult" remained in Providers.
1. In most cases, we should prioritize using Python’s standard exceptions (e.g., `ValueError`, `TypeError`, `OSError`) instead of wrapping everything in `AirflowException`. Yes 2. In `airflow-core`, we should define and use more specific subclasses under `airflow.exceptions`. Yes 3. For the providers, we should also define exceptions within `airflow.providers.<provider>.exceptions`. Yes if none of the 1. is appropriate (which should be a rarity). This should only be needed if we want to use some "custom" behaviour, property or otherwise "common" thing between the exceptions (for example aws exception could hold region, url and other things that could help to identify the failure better). - Should we remove the existing ones? (We previously had concerns about backward compatibility.) Or should we only apply this pattern to new pull requests? Yes. I would remove it completely with deprecation_tools as we do for other removed airflow modules and classes. It will make it invisible to IDE tools and autocompletion but old usages will still work. There is a bit of difficulty with removal, because currently others could catch `AirflowException` - so we would have to figure out how to make our "specific" exceptions to still derive from AirflowException that is missing. But I think we can work it out - by aliasing AirflowException to be something completely different (__InternalAirflowException) placed in a different module and suppressing the warning when they are used by our internal exception - or we could do some meta-classes magic to handle it. - Should we enforce it through development tools? If we remove the AirflowException - no enforcement will be needed. - Is there an acceptable exception for using `AirflowException`? Not that I am aware of in our code. But users could have used it and we should keep compatibility. On Wed, Aug 27, 2025 at 10:16 AM Wei Lee <weilee...@gmail.com> wrote: > Hi all, > > I would like to propose avoiding, or at least reducing, the direct use of > `AirflowException`. > > `AirflowException` is used extensively across our codebase, with the > following occurrences: > > - 54 in `airflow-core` > - 1,515 in `providers` > - 20 in `task-sdk` > > `AirflowException` itself does not provide much more information than a > standard `Exception`, aside from indicating that it originates from > Airflow. > > I would like to propose the following practices, combining ideas from our > previous discussion on Slack: > > 1. In most cases, we should prioritize using Python’s standard exceptions > (e.g., `ValueError`, `TypeError`, `OSError`) instead of wrapping everything > in `AirflowException`. > 2. In `airflow-core`, we should define and use more specific subclasses > under `airflow.exceptions`. > 3. For the providers, we should also define exceptions within > `airflow.providers.<provider>.exceptions`. > > If this approach makes sense, there are some following questions. > > - Should we remove the existing ones? (We previously had concerns about > backward compatibility.) Or should we only apply this pattern to new pull > requests? > - Should we enforce it through development tools? > - Is there an acceptable exception for using `AirflowException`? > > Best regards, > Wei Lee >
