Hi Sameer, Thanks for your thoughtful questions! Here are my thoughts on those points:
1) I see the value here mainly in terms of efficiency and sustainability. By computing a solution once and storing it, we can save the time and tokens an LLM would otherwise use to solve the same problem over and over. It's a nice way to be more environmentally friendly, too! 2) Since many contributions are already agent-assisted, we can address hallucinations by adding some guidance to AGENTS.md. Rather than relying on periodic checks, we can ask agents to perform their own consistency and hallucination checks during the PR process. This keeps a human naturally in the loop during review, turning token usage into a great opportunity for automated quality control—which we ask our contributors' agents to perform. This is actually a very interesting "twist" - because this also nicely approaches the "assymetry" we have by having a lot more AI assisted contributions. We can ask - in our AGENTS.md those agents to do **way** mor things to make our projects better - all of them done on the local machines of the contributors, with their tokens being spent. Best, Jarek Potiuk On Sun, May 31, 2026 at 4:55 PM Sameer Mesiah <[email protected]> wrote: > Hi Jarek, > > I do have a few questions about the "suggestions database" point: > > 1) Previously, it was agreed that since LLMs work well with unstructured > error messages, there was minimal benefit to an additional taxonomy layer > on top of existing exceptions. I was just hoping you could elaborate on the > marginal value of a structured data store over better exception messages? > > 2) If we were to implement this "suggestions database", do you believe the > hallucination rate of the current agentic systems we have available is low > enough that it requires minimal manual intervention to the extent that it > is worthwhile to introduce? If a maintainer/committer/contributor has to > skim through the database periodically to remove hallucinations, would we > not be expending tokens in favour of negligible time savings? > > Granted, I could be overestimating the hallucination rate of these agentic > systems (I have never used them personally) so it might be the ideal > solution. It's an interesting idea anyway so I would love to hear more > about it. > > On Sun, 31 May 2026 at 15:37, Jarek Potiuk <[email protected]> wrote: > > > Hi all, > > > > TP's suggestion aligns with Omkar’s earlier proposal ( > > https://lists.apache.org/thread/f275xo3tjd4olsmrc8nggncs62fjnl0x). I see > > two main approaches: > > > > 1. Generic Python exceptions: Simple and concise, but difficult for > > providing nuanced diagnostic or resolution guidance in logs. > > 2. Custom exception hierarchy: Allows us to accumulate specific > > troubleshooting documentation for users. This also enables the > non-breaking > > compatibility TP mentioned. > > > > While I previously favored simplicity, I now lean toward a structured > > hierarchy. In an agentic context, we could automate the maintenance of a > > "suggestions database" by scanning past issues. This would provide users > > (and agents) with clear, documented resolution steps, ultimately saving > > time and tokens. > > > > Best, > > Jarek Potiuk > > > > > > On Sun, May 31, 2026 at 4:19 PM Tzu-ping Chung via dev < > > [email protected]> wrote: > > > > > I think we can make this non-breaking entirely? We can simply raise a > > > subclass of both AirflowException and whatever we want to migrate to. > > > > > > If we want to, we can emit a deprecation warning if AirflowException is > > > imported directly by the user, but I feel even that is particularly > > needed. > > > > > > Raising a custom exception that’s as specific as practical is always > the > > > best practice. Using a custom exception that subclasses both built-in > and > > > AirflowException (but not AirflowException itself) is a good idea on > its > > > own. > > > > > > TP > > > > > > > > > > On 31 May 2026, at 21:25, Jens Scheffler <[email protected]> > wrote: > > > > > > > > Hi, > > > > > > > > I do not have a stron opinion on this, with the past decision we > mainly > > > wanted to ensure code is clean. > > > > > > > > For me also Option (3) would be viable: We leave existing code as it > is > > > and just ensure that new code is clean. But I am also okay to join the > > > other majority opinion. > > > > > > > > Jens > > > > > > > > On 31.05.26 14:08, Sameer Mesiah wrote: > > > >> Hi, > > > >> > > > >> Thanks for opening this thread. This was long overdue considering > that > > > >> current migration efforts for AirflowException are more scattered > > rather > > > >> than a consolidated pre-planned wave. > > > >> > > > >> In general, I agree with your overall approach. But there are a few > > > things > > > >> I want to raise in response to your proposal: > > > >> > > > >> On 1) > > > >> > > > >> I think Amogh already expanded on thiss somewhat. But I believe it > is > > > >> important to understand that AirflowException is not constrained to > > > >> the provider layer but often bubble up from Airflow core.For example > > > >> task_runner.py explicitly catches AirflowException as part of task > > > >> execution and retry handling, and there are also usages in the hook > > and > > > >> connection layers. > > > >> > > > >> But depending on whether we scope this to core + providers vs > > providers > > > >> only, the migration strategy looks quite different. If the > intention > > is > > > >> provider-only migration, users will continue to encounter > > > AirflowException > > > >> from core Airflow components, which may make the overall exception > > model > > > >> feel somewhat inconsistent. If the intention is project-wide > > > deprecation, I > > > >> think we need a broader plan than provider-level migrations alone. > > > >> > > > >> Also, as Amogh already implied (he is free to correct me in case I > > > >> misinterpreted his reservations), deprecation warnings may not be > > > >> worthwhile here. Unlike a deprecated API or method, this is not > > > something > > > >> users are actively choosing to call. In most cases, Airflow itself > is > > > >> deciding which exception type to raise. I am leaning against > warnings > > > due > > > >> to the potential for log pollution and the relatively limited value > > they > > > >> provide. > > > >> > > > >> On 2) > > > >> > > > >> I agree with your per-package migration proposal. But as Amogh said, > > we > > > >> should ensure new exceptions inherit from AirflowException to > prevent > > > >> breaking changes across the provider ecosystem. I must add that > there > > > will > > > >> inevitably be a period where users run multiple providers, some > > migrated > > > >> and some not. This may result in a mixture of exception types > > appearing > > > in > > > >> logs and stack traces. While not ideal, this is probably unavoidable > > > given > > > >> independent provider versioning. As long as migrations are clearly > > > >> documented, I do not see this as a major issue. > > > >> > > > >> My answers to your questions: > > > >> > > > >> *Should we add a deprecation warning to AirflowException?* > > > >> > > > >> I am +0. Not strongly opposed, but I am leaning no due to the risk > of > > > log > > > >> pollution and the fact that the warning would be emitted by Airflow > > > itself > > > >> rather than as a result of an explicit user action. > > > >> > > > >> *Is the per-package migration approach reasonable?* > > > >> > > > >> Yes. No blocking concerns from my side. > > > >> > > > >> *Should we define a project-wide exception taxonomy, or leave > > exception > > > >> choices to individual provider teams?* > > > >> > > > >> I would leave this to individual provider teams. The providers have > > very > > > >> different APIs, error models, and opinions around exception > handling. > > > >> Attempting to enforce a central taxonomy across all providers may be > > > >> impractical and may not provide much practical value over what we > > > currently > > > >> have in place. > > > >> > > > >> Thanks, > > > >> Sameer Mesiah. > > > >> > > > >> On Sat, 30 May 2026 at 10:59, Shahar Epstein <[email protected]> > > wrote: > > > >> > > > >>> Hi all, > > > >>> > > > >>> This follows up on the earlier lazy-consensus > > > >>> <https://lists.apache.org/thread/m86gs4mqt8hq1n26g0pp0fq5h5g0x2q9> > > and > > > >>> related discussions around deprecating AirflowException. > > > >>> > > > >>> While working on a cleanup PR for the Google Cloud Run operator in > > the > > > >>> Google provider (#67769), Amogh pointed out that replacing > > > AirflowException > > > >>> with built-in or custom exceptions is a breaking change. Users may > > > rely on > > > >>> AirflowException in callbacks (on_failure_callback, > > on_retry_callback) > > > or > > > >>> in custom wrappers that catch it explicitly. > > > >>> > > > >>> Rather than making these changes incrementally across individual > PRs, > > > I'd > > > >>> like to discuss a coordinated migration strategy. > > > >>> Proposal > > > >>> > > > >>> *1. Emit a deprecation warning from **AirflowException.__init__**.* > > > >>> > > > >>> Although emitting warnings when exceptions are raised is uncommon, > it > > > is > > > >>> justified here because users may depend on AirflowException in > > > callbacks > > > >>> and exception handlers. This provides advance notice before > > > >>> package-specific migrations occur. > > > >>> > > > >>> *2. Migrate per package in the next breaking release of that > > package.* > > > >>> > > > >>> Require a single migration PR per package (airflow-core and each > > > provider), > > > >>> rather than spreading exception changes across multiple PRs and > > > releases. > > > >>> The migration would be included in that package's next breaking > > > release and > > > >>> documented accordingly. > > > >>> Questions > > > >>> > > > >>> - Should we add a deprecation warning to AirflowException? > > > >>> - Is the per-package migration approach reasonable? > > > >>> - Should we define a project-wide exception taxonomy, or leave > > > exception > > > >>> choices to individual provider teams? > > > >>> > > > >>> If there's agreement, I'll follow up with a lazy-consensus > proposal. > > > >>> > > > >>> Thanks, > > > >>> > > > >>> > > > >>> Shahar > > > >>> > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > >
