David - It is specificity. `AirflowException` is a catch-all exception and whenever it is raised, the caller has no information about what went wrong without parsing the message string. An exception such as `ConnectionTimeoutError` on the other hand tells you immediately what happened and lets you handle different failure modes differently in code.
You cannot write `except AirflowException` and meaningfully distinguish a transient network failure from a permanent config error without inspecting the message On the contrary, builtins like ValueError, ConnectionError, TimeoutError integrate naturally with standard python tooling and are familiar to users who are not deeply involved Airflow users and to add, provider-specific exception types give provider maintainers a clean(er) surface to document resolution steps The goal is not to do this overnight but to stop using it as a default when something more specific is appropriate and/or available. Thanks & Regards, Amogh Desai On Mon, Jun 1, 2026 at 5:22 PM Blain David <[email protected]> wrote: > Probably a stupid question from my side, but why do we want to migrate > away from AirflowException? > > ________________________________ > From: Jarek Potiuk <[email protected]> > Sent: Monday, June 01, 2026 10:48 > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] Migrating from AirflowException to built-in / > custom exceptions > > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze > niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, > stuur deze e-mail als bijlage naar [email protected]<mailto: > [email protected]>. > > - Where does the database live and who owns it? Is this a new repo, part of > the docs, something in CLAUDE.md? > > Just generated docs in airflow.apache.org where the errors point at. No > "database" per-se > > - What does the write path look like? Agents write during PR review, but > what would be the trigger? any new exception type, > any failing test, or something else? > > Just regular PR updates and AGENTS.md instructing to reconcile it whenever > something about exceptions change. > > On Mon, Jun 1, 2026 at 7:56 AM Amogh Desai <[email protected]> wrote: > > > I like TP's dual-subclass approach. I see it as the most well-scoped and > > the cleanest solution here and I think it > > resolves most of the open questions. Raise a specific exception that > > inherits from both AirflowException and the > > target type, the existing catch based code will keep working, new code > gets > > specificity, no deprecation cycles or windows > > to manage. > > > > On Jarek's suggestions database idea, I think that's worth a separate > > thread. The exception hierarchy should not be designed > > around a storage/retrieval system that does not exist yet. And the > database > > idea does not actually depend on a new exception > > hierarchy either, it could work off existing error messages. Conflating > the > > two makes both harder to evaluate. > > > > Jarek, a couple of questions if you want to push the suggestions database > > idea forward: > > > > - Where does the database live and who owns it? Is this a new repo, part > of > > the docs, something in CLAUDE.md? > > - What does the write path look like? Agents write during PR review, but > > what would be the trigger? any new exception type, > > any failing test, or something else? > > > > Happy to ask further details and see it in a different proposal if we > have > > appetite at the moment. > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Sun, May 31, 2026 at 10:07 PM Jarek Potiuk <[email protected]> wrote: > > > > > > Okay. That sounds great. It would be interesting to see what the > system > > > > > > Well.. If others agree that is :). That is just one of the options :) > > > > > > On Sun, May 31, 2026 at 6:23 PM Sameer Mesiah <[email protected]> > > > wrote: > > > > > > > Okay. That sounds great. It would be interesting to see what the > system > > > > looks like when it is live. > > > > > > > > Thanks, > > > > Sameer Mesiah. > > > > > > > > On Sun, 31 May 2026 at 17:15, Jarek Potiuk <[email protected]> wrote: > > > > > > > > > Hi Sameer, > > > > > > > > > > Thanks for your thoughtful questions! Here are my thoughts on those > > > > points: > > > > > > > > > > 1) I see the value here mainly in terms of efficiency and > > > > sustainability. > > > > > By computing a solution once and storing it, we can save the time > and > > > > > tokens an LLM would otherwise use to solve the same problem over > and > > > > over. > > > > > It's a nice way to be more environmentally friendly, too! > > > > > > > > > > 2) Since many contributions are already agent-assisted, we can > > address > > > > > hallucinations by adding some guidance to AGENTS.md. Rather than > > > relying > > > > on > > > > > periodic checks, we can ask agents to perform their own consistency > > and > > > > > hallucination checks during the PR process. This keeps a human > > > naturally > > > > in > > > > > the loop during review, turning token usage into a great > opportunity > > > for > > > > > automated quality control—which we ask our contributors' agents to > > > > perform. > > > > > > > > > > This is actually a very interesting "twist" - because this also > > nicely > > > > > approaches the "assymetry" we have by having a lot more AI assisted > > > > > contributions. We can ask - in our AGENTS.md those agents to do > > **way** > > > > mor > > > > > things to make our projects better - all of them done on the local > > > > machines > > > > > of the contributors, with their tokens being spent. > > > > > > > > > > Best, > > > > > Jarek Potiuk > > > > > > > > > > > > > > > On Sun, May 31, 2026 at 4:55 PM Sameer Mesiah < > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi Jarek, > > > > > > > > > > > > I do have a few questions about the "suggestions database" point: > > > > > > > > > > > > 1) Previously, it was agreed that since LLMs work well with > > > > unstructured > > > > > > error messages, there was minimal benefit to an additional > taxonomy > > > > layer > > > > > > on top of existing exceptions. I was just hoping you could > > elaborate > > > on > > > > > the > > > > > > marginal value of a structured data store over better exception > > > > messages? > > > > > > > > > > > > 2) If we were to implement this "suggestions database", do you > > > believe > > > > > the > > > > > > hallucination rate of the current agentic systems we have > available > > > is > > > > > low > > > > > > enough that it requires minimal manual intervention to the extent > > > that > > > > it > > > > > > is worthwhile to introduce? If a maintainer/committer/contributor > > has > > > > to > > > > > > skim through the database periodically to remove hallucinations, > > > would > > > > we > > > > > > not be expending tokens in favour of negligible time savings? > > > > > > > > > > > > Granted, I could be overestimating the hallucination rate of > these > > > > > agentic > > > > > > systems (I have never used them personally) so it might be the > > ideal > > > > > > solution. It's an interesting idea anyway so I would love to hear > > > more > > > > > > about it. > > > > > > > > > > > > On Sun, 31 May 2026 at 15:37, Jarek Potiuk <[email protected]> > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > TP's suggestion aligns with Omkar’s earlier proposal ( > > > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Ff275xo3tjd4olsmrc8nggncs62fjnl0x&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf89431ab18864bb78cf808debfba8eae%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639159005223102938%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=8N88zAIQMIBsIQQYUei89ZlY30vpdbOylnl%2BUbZiFjQ%3D&reserved=0 > <https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x> > > ). > > > I > > > > > see > > > > > > > two main approaches: > > > > > > > > > > > > > > 1. Generic Python exceptions: Simple and concise, but > difficult > > > for > > > > > > > providing nuanced diagnostic or resolution guidance in logs. > > > > > > > 2. Custom exception hierarchy: Allows us to accumulate > specific > > > > > > > troubleshooting documentation for users. This also enables the > > > > > > non-breaking > > > > > > > compatibility TP mentioned. > > > > > > > > > > > > > > While I previously favored simplicity, I now lean toward a > > > structured > > > > > > > hierarchy. In an agentic context, we could automate the > > maintenance > > > > of > > > > > a > > > > > > > "suggestions database" by scanning past issues. This would > > provide > > > > > users > > > > > > > (and agents) with clear, documented resolution steps, > ultimately > > > > saving > > > > > > > time and tokens. > > > > > > > > > > > > > > Best, > > > > > > > Jarek Potiuk > > > > > > > > > > > > > > > > > > > > > On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > I think we can make this non-breaking entirely? We can simply > > > > raise a > > > > > > > > subclass of both AirflowException and whatever we want to > > migrate > > > > to. > > > > > > > > > > > > > > > > If we want to, we can emit a deprecation warning if > > > > AirflowException > > > > > is > > > > > > > > imported directly by the user, but I feel even that is > > > particularly > > > > > > > needed. > > > > > > > > > > > > > > > > Raising a custom exception that’s as specific as practical is > > > > always > > > > > > the > > > > > > > > best practice. Using a custom exception that subclasses both > > > > built-in > > > > > > and > > > > > > > > AirflowException (but not AirflowException itself) is a good > > idea > > > > on > > > > > > its > > > > > > > > own. > > > > > > > > > > > > > > > > TP > > > > > > > > > > > > > > > > > > > > > > > > > On 31 May 2026, at 21:25, Jens Scheffler < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I do not have a stron opinion on this, with the past > decision > > > we > > > > > > mainly > > > > > > > > wanted to ensure code is clean. > > > > > > > > > > > > > > > > > > For me also Option (3) would be viable: We leave existing > > code > > > as > > > > > it > > > > > > is > > > > > > > > and just ensure that new code is clean. But I am also okay to > > > join > > > > > the > > > > > > > > other majority opinion. > > > > > > > > > > > > > > > > > > Jens > > > > > > > > > > > > > > > > > > On 31.05.26 14:08, Sameer Mesiah wrote: > > > > > > > > >> Hi, > > > > > > > > >> > > > > > > > > >> Thanks for opening this thread. This was long overdue > > > > considering > > > > > > that > > > > > > > > >> current migration efforts for AirflowException are more > > > > scattered > > > > > > > rather > > > > > > > > >> than a consolidated pre-planned wave. > > > > > > > > >> > > > > > > > > >> In general, I agree with your overall approach. But there > > are > > > a > > > > > few > > > > > > > > things > > > > > > > > >> I want to raise in response to your proposal: > > > > > > > > >> > > > > > > > > >> On 1) > > > > > > > > >> > > > > > > > > >> I think Amogh already expanded on thiss somewhat. But I > > > believe > > > > it > > > > > > is > > > > > > > > >> important to understand that AirflowException is not > > > constrained > > > > > to > > > > > > > > >> the provider layer but often bubble up from Airflow > core.For > > > > > example > > > > > > > > >> task_runner.py explicitly catches AirflowException as part > > of > > > > task > > > > > > > > >> execution and retry handling, and there are also usages in > > the > > > > > hook > > > > > > > and > > > > > > > > >> connection layers. > > > > > > > > >> > > > > > > > > >> But depending on whether we scope this to core + providers > > vs > > > > > > > providers > > > > > > > > >> only, the migration strategy looks quite different. If > the > > > > > > intention > > > > > > > is > > > > > > > > >> provider-only migration, users will continue to encounter > > > > > > > > AirflowException > > > > > > > > >> from core Airflow components, which may make the overall > > > > exception > > > > > > > model > > > > > > > > >> feel somewhat inconsistent. If the intention is > project-wide > > > > > > > > deprecation, I > > > > > > > > >> think we need a broader plan than provider-level > migrations > > > > alone. > > > > > > > > >> > > > > > > > > >> Also, as Amogh already implied (he is free to correct me > in > > > > case I > > > > > > > > >> misinterpreted his reservations), deprecation warnings may > > not > > > > be > > > > > > > > >> worthwhile here. Unlike a deprecated API or method, this > is > > > not > > > > > > > > something > > > > > > > > >> users are actively choosing to call. In most cases, > Airflow > > > > itself > > > > > > is > > > > > > > > >> deciding which exception type to raise. I am leaning > against > > > > > > warnings > > > > > > > > due > > > > > > > > >> to the potential for log pollution and the relatively > > limited > > > > > value > > > > > > > they > > > > > > > > >> provide. > > > > > > > > >> > > > > > > > > >> On 2) > > > > > > > > >> > > > > > > > > >> I agree with your per-package migration proposal. But as > > Amogh > > > > > said, > > > > > > > we > > > > > > > > >> should ensure new exceptions inherit from AirflowException > > to > > > > > > prevent > > > > > > > > >> breaking changes across the provider ecosystem. I must add > > > that > > > > > > there > > > > > > > > will > > > > > > > > >> inevitably be a period where users run multiple providers, > > > some > > > > > > > migrated > > > > > > > > >> and some not. This may result in a mixture of exception > > types > > > > > > > appearing > > > > > > > > in > > > > > > > > >> logs and stack traces. While not ideal, this is probably > > > > > unavoidable > > > > > > > > given > > > > > > > > >> independent provider versioning. As long as migrations are > > > > clearly > > > > > > > > >> documented, I do not see this as a major issue. > > > > > > > > >> > > > > > > > > >> My answers to your questions: > > > > > > > > >> > > > > > > > > >> *Should we add a deprecation warning to > AirflowException?* > > > > > > > > >> > > > > > > > > >> I am +0. Not strongly opposed, but I am leaning no due to > > the > > > > risk > > > > > > of > > > > > > > > log > > > > > > > > >> pollution and the fact that the warning would be emitted > by > > > > > Airflow > > > > > > > > itself > > > > > > > > >> rather than as a result of an explicit user action. > > > > > > > > >> > > > > > > > > >> *Is the per-package migration approach reasonable?* > > > > > > > > >> > > > > > > > > >> Yes. No blocking concerns from my side. > > > > > > > > >> > > > > > > > > >> *Should we define a project-wide exception taxonomy, or > > leave > > > > > > > exception > > > > > > > > >> choices to individual provider teams?* > > > > > > > > >> > > > > > > > > >> I would leave this to individual provider teams. The > > providers > > > > > have > > > > > > > very > > > > > > > > >> different APIs, error models, and opinions around > exception > > > > > > handling. > > > > > > > > >> Attempting to enforce a central taxonomy across all > > providers > > > > may > > > > > be > > > > > > > > >> impractical and may not provide much practical value over > > what > > > > we > > > > > > > > currently > > > > > > > > >> have in place. > > > > > > > > >> > > > > > > > > >> Thanks, > > > > > > > > >> Sameer Mesiah. > > > > > > > > >> > > > > > > > > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >>> Hi all, > > > > > > > > >>> > > > > > > > > >>> This follows up on the earlier lazy-consensus > > > > > > > > >>> < > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm86gs4mqt8hq1n26g0pp0fq5h5g0x2q9&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf89431ab18864bb78cf808debfba8eae%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639159005223124368%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=LqVG4rOvDCEQKE%2BJ3TCzkzlum220KN%2F50jethHYsVGo%3D&reserved=0 > ><https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9> > > > > > > > and > > > > > > > > >>> related discussions around deprecating AirflowException. > > > > > > > > >>> > > > > > > > > >>> While working on a cleanup PR for the Google Cloud Run > > > operator > > > > > in > > > > > > > the > > > > > > > > >>> Google provider (#67769), Amogh pointed out that > replacing > > > > > > > > AirflowException > > > > > > > > >>> with built-in or custom exceptions is a breaking change. > > > Users > > > > > may > > > > > > > > rely on > > > > > > > > >>> AirflowException in callbacks (on_failure_callback, > > > > > > > on_retry_callback) > > > > > > > > or > > > > > > > > >>> in custom wrappers that catch it explicitly. > > > > > > > > >>> > > > > > > > > >>> Rather than making these changes incrementally across > > > > individual > > > > > > PRs, > > > > > > > > I'd > > > > > > > > >>> like to discuss a coordinated migration strategy. > > > > > > > > >>> Proposal > > > > > > > > >>> > > > > > > > > >>> *1. Emit a deprecation warning from > > > > > **AirflowException.__init__**.* > > > > > > > > >>> > > > > > > > > >>> Although emitting warnings when exceptions are raised is > > > > > uncommon, > > > > > > it > > > > > > > > is > > > > > > > > >>> justified here because users may depend on > AirflowException > > > in > > > > > > > > callbacks > > > > > > > > >>> and exception handlers. This provides advance notice > before > > > > > > > > >>> package-specific migrations occur. > > > > > > > > >>> > > > > > > > > >>> *2. Migrate per package in the next breaking release of > > that > > > > > > > package.* > > > > > > > > >>> > > > > > > > > >>> Require a single migration PR per package (airflow-core > and > > > > each > > > > > > > > provider), > > > > > > > > >>> rather than spreading exception changes across multiple > PRs > > > and > > > > > > > > releases. > > > > > > > > >>> The migration would be included in that package's next > > > breaking > > > > > > > > release and > > > > > > > > >>> documented accordingly. > > > > > > > > >>> Questions > > > > > > > > >>> > > > > > > > > >>> - Should we add a deprecation warning to > > AirflowException? > > > > > > > > >>> - Is the per-package migration approach reasonable? > > > > > > > > >>> - Should we define a project-wide exception taxonomy, > or > > > > leave > > > > > > > > exception > > > > > > > > >>> choices to individual provider teams? > > > > > > > > >>> > > > > > > > > >>> If there's agreement, I'll follow up with a > lazy-consensus > > > > > > proposal. > > > > > > > > >>> > > > > > > > > >>> Thanks, > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> Shahar > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > For additional commands, e-mail: > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
