This is an automated email from the ASF dual-hosted git repository.

Lee-W pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new 5a0a26f90ea Show common.ai retry policies why each fallback connection 
failed (#76999)
5a0a26f90ea is described below

commit 5a0a26f90ea24be037e5d5b779e46752dd9853d2
Author: Wei Lee <[email protected]>
AuthorDate: Wed Nov 18 22:12:16 2026 +0800

    Show common.ai retry policies why each fallback connection failed (#76999)
---
 providers/common/ai/docs/changelog.rst             |   9 ++
 providers/common/ai/docs/provider_fallback.rst     |  26 +++-
 providers/common/ai/docs/recovery.rst              |  30 ++--
 providers/common/ai/docs/retry_policies.rst        |  41 ++++--
 .../airflow/providers/common/ai/durable/journal.py |  49 +++++--
 .../providers/common/ai/fallback_exceptions.py     |  65 +++++++++
 .../providers/common/ai/mixins/cancellable_run.py  |  21 ++-
 .../airflow/providers/common/ai/policies/retry.py  |  77 +++++++---
 .../tests/unit/common/ai/durable/test_journal.py   |  42 ++++++
 .../unit/common/ai/mixins/test_cancellable_run.py  | 129 ++++++++++++++++-
 .../ai/tests/unit/common/ai/policies/test_retry.py | 157 ++++++++++++++++++++-
 .../unit/common/ai/test_fallback_exceptions.py     |  42 ++++++
 12 files changed, 634 insertions(+), 54 deletions(-)

diff --git a/providers/common/ai/docs/changelog.rst 
b/providers/common/ai/docs/changelog.rst
index fab49897489..f3281ab5c1f 100644
--- a/providers/common/ai/docs/changelog.rst
+++ b/providers/common/ai/docs/changelog.rst
@@ -25,6 +25,15 @@
 Changelog
 ---------
 
+.. note::
+  When every connection of a ``fallback_conn_ids`` chain fails, 
``LLMOperator`` and ``AgentOperator``
+  now raise 
``airflow.providers.common.ai.fallback_exceptions.FallbackChainExhaustedError`` 
instead of
+  pydantic-ai's ``FallbackExceptionGroup``. It subclasses 
``FallbackExceptionGroup``, so ``except``
+  clauses and ``RetryRule(exception=FallbackExceptionGroup)`` keep matching, 
and ``__cause__`` is the
+  group the chain raised. ``LLMRetryPolicy`` and ``ClassifierRetryPolicy`` now 
also match their
+  ``fallback_rules`` against the last connection's error, and show the model 
each connection's error
+  and how many durable journal steps the attempt replayed and recorded.
+
 .. note::
   ``@task.llm``, ``@task.agent``, ``@task.llm_sql``, ``@task.llm_branch``,
   ``@task.llm_file_analysis`` and ``@task.llm_schema_compare`` now render only 
the prompt returned
diff --git a/providers/common/ai/docs/provider_fallback.rst 
b/providers/common/ai/docs/provider_fallback.rst
index 3dc2595f64a..11823b06b2e 100644
--- a/providers/common/ai/docs/provider_fallback.rst
+++ b/providers/common/ai/docs/provider_fallback.rst
@@ -140,11 +140,27 @@ vendor can answer the same prompt now; a bad prompt, an 
exhausted quota on every
 a permanent data error still has to be decided by the retry policy.
 
 Adding a chain changes what the retry layer sees. When every connection in the 
chain
-fails, the exception the task raises is 
``pydantic_ai.exceptions.FallbackExceptionGroup``,
-not the last provider's own exception, so retry rules matched against a 
provider-specific
-exception type stop matching. Before adding a chain to a connection that Dags 
already use,
-read :doc:`retry_policies` -- the section "When the connection also carries a 
fallback
-chain" spells out what to check.
+fails, the task raises a ``pydantic_ai.exceptions.FallbackExceptionGroup``, 
not the last
+provider's own exception. Which one depends on how the agent runs:
+
+* ``LLMOperator``, ``AgentOperator``, their subclasses, and the matching 
``@task.llm`` and
+  ``@task.agent`` decorators raise
+  
:class:`~airflow.providers.common.ai.fallback_exceptions.FallbackChainExhaustedError`,
 a
+  subclass of ``FallbackExceptionGroup``. The error each connection raised is 
in its
+  ``member_errors``, and with ``durable=True`` its ``replayed_steps`` and 
``recorded_steps``
+  say how far the attempt got.
+* In a plain ``@task`` that builds an agent with ``PydanticAIHook`` and runs 
it, the task
+  raises pydantic-ai's own ``FallbackExceptionGroup``. It has no 
``member_errors``,
+  ``replayed_steps`` or ``recorded_steps``; read each connection's error from 
the group's
+  ``.exceptions``. An ``except FallbackChainExhaustedError`` there never 
matches.
+
+On both paths ``LLMRetryPolicy`` and ``ClassifierRetryPolicy`` match their
+``fallback_rules`` against the chain's error and then against the last 
connection's error,
+so a rule written for a provider-specific exception type keeps matching. A 
plain
+``ExceptionRetryPolicy``, or any other policy that compares one exception 
type, sees only
+the group, and a rule written for a provider-specific exception type stops 
matching there.
+Before adding a chain to a connection that Dags already use, read 
:doc:`retry_policies` --
+the section "When the connection also carries a fallback chain" spells out 
what to check.
 
 Costs to know before configuring a long chain
 ---------------------------------------------
diff --git a/providers/common/ai/docs/recovery.rst 
b/providers/common/ai/docs/recovery.rst
index 9e0c83bd85f..5ce975973ec 100644
--- a/providers/common/ai/docs/recovery.rst
+++ b/providers/common/ai/docs/recovery.rst
@@ -42,7 +42,8 @@ What each failure costs
      - Kept. The next connection receives the run's message history, including 
every
        tool result, and continues the run.
    * - Every connection in the chain fails, and the task raises
-       ``pydantic_ai.exceptions.FallbackExceptionGroup``
+       ``FallbackChainExhaustedError`` (a 
``pydantic_ai.exceptions.FallbackExceptionGroup``;
+       see :doc:`provider_fallback` for which tasks raise it)
      - Airflow ``retries`` with ``durable=True``
      - Replayed. The retry replays the steps an earlier attempt recorded, 
whichever
        connection answered them, and continues on the first connection that 
answers now.
@@ -125,10 +126,12 @@ Retries after the whole chain fails
 -----------------------------------
 
 With ``durable=True``, each step is recorded in the task instance's journal 
whichever
-connection answered it. When every connection fails, the task raises
-``FallbackExceptionGroup`` and Airflow retries it. The retry replays the 
recorded steps,
-then sends its first live request through the chain from the top. The run 
continues on
-the first connection that answers, whether or not that is the one that 
answered before.
+connection answered it. When every connection fails, the operator (or 
``@task.llm`` /
+``@task.agent``) raises ``FallbackChainExhaustedError``, and a plain ``@task`` 
with
+``AirflowDurability`` raises the bare ``FallbackExceptionGroup`` (see
+:doc:`provider_fallback`). Airflow retries either one. The retry replays the 
recorded steps,
+then sends its first live request through the chain from the top. The run 
continues on the
+first connection that answers, whether or not that is the one that answered 
before.
 
 A recorded model step replays only if the retry's request has the same 
fingerprint, and
 for a chain the model in the fingerprint is the whole chain, whichever 
connection
@@ -144,11 +147,18 @@ What is not covered
   whose worker died before its result was recorded. Make such tools 
idempotent; see
   "Side effects and idempotency" in :doc:`durable_execution`.
 - A request every provider rejects, such as a malformed one that each returns 
a 400
-  for, walks the whole chain on every attempt. A ``RetryRule`` cannot stop 
those retries
-  without stopping outage retries too: once a chain is configured, both raise
-  ``FallbackExceptionGroup``, and a rule compares only that top-level type. 
Retry
-  policies need Airflow 3.3 or later; see "When the connection also carries a 
fallback
-  chain" in :doc:`retry_policies`.
+  for, walks the whole chain on every attempt. The error the task raises is a
+  ``FallbackExceptionGroup``: ``FallbackChainExhaustedError``, carrying each 
connection's
+  error in ``member_errors``, from the operators and decorators, and the plain 
group from a
+  hand-built agent (see :doc:`provider_fallback`). ``LLMRetryPolicy`` and
+  ``ClassifierRetryPolicy`` match their ``fallback_rules`` against the last 
connection's
+  error too, so a rule can fail a permanent error such as a bad request 
without failing
+  outage retries, provided the two raise different exception types. Telling a 
rate limit
+  from a bad request needs ``ModelRateLimitError`` and 
``ModelRequestRejectedError``
+  rules; see :doc:`retry_policies` for the pydantic-ai-slim versions that have 
them. A plain
+  ``ExceptionRetryPolicy`` compares only the group, and a retry still counts 
against
+  ``retries``. Retry policies need Airflow 3.3 or later; see "When the 
connection also
+  carries a fallback chain" in :doc:`retry_policies`.
 - No connection type builds the ``AnthropicVertex`` client that Claude on 
Vertex AI
   needs, so a chain cannot include Claude on Vertex AI. A Vertex AI connection 
reaches
   Gemini.
diff --git a/providers/common/ai/docs/retry_policies.rst 
b/providers/common/ai/docs/retry_policies.rst
index e8eeab44a46..b5567cfb8cb 100644
--- a/providers/common/ai/docs/retry_policies.rst
+++ b/providers/common/ai/docs/retry_policies.rst
@@ -379,15 +379,38 @@ retry behaviour with no change to the Dag. Two things 
follow:
 Separately, and regardless of this policy: if the connection **the task 
itself** uses to call
 the LLM (for example ``llm_conn_id`` on ``LLMOperator`` or ``AgentOperator``) 
carries a fallback
 chain, the exception the task raises once that chain is exhausted is
-``pydantic_ai.exceptions.FallbackExceptionGroup``, not the last provider's own 
exception.
-``RetryRule`` matches with ``isinstance``, so a rule written as
-``RetryRule(exception=ModelHTTPError, ...)`` -- in ``fallback_rules`` here or 
in a plain
-``ExceptionRetryPolicy`` -- stops matching. Match 
``pydantic_ai.exceptions.FallbackExceptionGroup``
-explicitly as well; its only common ancestor with ``ModelAPIError`` is 
``Exception``, too
-broad to write a rule against. The original per-model exceptions are still 
available on
-``FallbackExceptionGroup.exceptions``, but ``RetryRule`` only compares the 
top-level
-exception type, so a rule set that told 429s apart from 400s collapses into 
one rule once
-the chain is in play.
+:class:`~airflow.providers.common.ai.fallback_exceptions.FallbackChainExhaustedError`.
 It subclasses
+``pydantic_ai.exceptions.FallbackExceptionGroup``, so a rule or ``except`` 
clause written for the
+group keeps matching, and it is raised ``from`` the group the chain raised. 
The error each
+connection raised is in ``member_errors``, in chain order. This holds for the 
operators, their
+subclasses and the ``@task.llm`` and ``@task.agent`` decorators; a plain 
``@task`` that runs an
+agent built with ``PydanticAIHook`` raises the bare 
``FallbackExceptionGroup``, without
+``member_errors``, and the rules below apply to it the same way (see 
:doc:`provider_fallback`).
+
+``RetryRule`` still compares one exception type, so the policies here read the 
members for it:
+in ``fallback_rules``, each rule is tried against the chain's own error and 
then against the
+error of the **last** connection, rule by rule in the order listed. A rule 
written as
+``RetryRule(exception=ModelHTTPError, ...)`` therefore matches again, and a 
rule set that tells
+failures apart by exception type, such as retrying a rate-limit error but 
failing a bad-request
+error, keeps telling them apart. To separate those two, name 
``ModelRateLimitError`` (from
+pydantic-ai-slim 2.59.0) and ``ModelRequestRejectedError`` (from 2.62.0) in 
the rules. This
+provider supports pydantic-ai-slim from 2.46.0. Before 2.59.0 a 429 is also a 
plain
+``ModelHTTPError``; from 2.59.0 to 2.61.x a rule can single out a rate limit 
but not a
+rejected request. When the connections failed with different errors, the last 
one decides;
+an earlier connection's rate limit does not make a final bad request 
retryable, and the
+reverse. A decision taken on the last connection's error says so in its 
reason. A rule
+cannot tell two failures of the same type apart, such as two 
``ModelHTTPError`` with
+different status codes; that needs a classifier. Only these policies look at 
the members: a
+plain ``ExceptionRetryPolicy`` still compares the chain's own error, so match
+``FallbackExceptionGroup`` there.
+
+The model that classifies sees the same detail. The prompt lists each 
connection's error by
+type and message, redacted like the rest of the error and limited to 
``max_exception_length``
+characters each, so a long message from one connection does not push the last 
one out, and,
+when the attempt kept a durable journal, how many steps the failed attempt 
replayed and
+recorded (``replayed_steps`` and ``recorded_steps`` on the error, which only 
the operator
+path sets). A retry still counts as an attempt against ``retries``: a policy 
can choose a
+longer delay or a different category from that detail, but cannot make a retry 
free.
 
 What the model can and cannot do
 --------------------------------
diff --git 
a/providers/common/ai/src/airflow/providers/common/ai/durable/journal.py 
b/providers/common/ai/src/airflow/providers/common/ai/durable/journal.py
index a3d7ad13bb1..77ce4e22c89 100644
--- a/providers/common/ai/src/airflow/providers/common/ai/durable/journal.py
+++ b/providers/common/ai/src/airflow/providers/common/ai/durable/journal.py
@@ -1069,6 +1069,40 @@ def journal_scope(journal: DurableJournal) -> 
Iterator[DurableJournal]:
         _ACTIVE_JOURNAL.reset(token)
 
 
+def _get_task_context_and_attempt() -> tuple[Any, str] | None:
+    """Return the running task's context and attempt key, or ``None`` outside 
a task."""
+    try:
+        context = get_current_context()
+    except (RuntimeError, AirflowException):
+        # Airflow 3 raises RuntimeError outside a task, Airflow 2 
AirflowException.
+        return None
+    return context, make_task_instance_run_key(context["task_instance"])
+
+
+def _find_attempt_journal(attempt: str) -> DurableJournal | None:
+    """Return the cached journal if it belongs to ``attempt``, else 
``None``."""
+    if _task_journal is not None and _task_journal[0] == attempt:
+        return _task_journal[1]
+    return None
+
+
+def find_active_journal() -> DurableJournal | None:
+    """
+    Return the journal already recording this run, without creating one.
+
+    A journal set with :func:`journal_scope` wins. Otherwise, inside a running 
task, this
+    returns the task attempt's journal if something already created it. 
``None`` means no
+    journal is recording: outside a task, or inside one where nothing has used 
durable
+    execution yet.
+    """
+    if (journal := _ACTIVE_JOURNAL.get()) is not None:
+        return journal
+    if (found := _get_task_context_and_attempt()) is None:
+        return None
+    _, attempt = found
+    return _find_attempt_journal(attempt)
+
+
 def current_journal() -> DurableJournal | None:
     """
     Return the active journal, or ``None`` outside an Airflow task.
@@ -1076,18 +1110,17 @@ def current_journal() -> DurableJournal | None:
     A journal set with :func:`journal_scope` wins. Otherwise, inside a running 
task,
     this returns the task attempt's own journal, created on first use, which 
deletes the
     steps of each run as soon as it succeeds. Outside a task there is nothing 
to replay
-    into, so durable execution is off and the agent runs as it would without 
it.
+    into, so durable execution is off and the agent runs as it would without 
it. To read
+    the journal without creating it, use :func:`find_active_journal`.
     """
     global _task_journal
 
     if (journal := _ACTIVE_JOURNAL.get()) is not None:
         return journal
-    try:
-        context = get_current_context()
-    except (RuntimeError, AirflowException):
-        # Airflow 3 raises RuntimeError outside a task, Airflow 2 
AirflowException.
+    if (found := _get_task_context_and_attempt()) is None:
         return None
-    attempt = make_task_instance_run_key(context["task_instance"])
-    if _task_journal is None or _task_journal[0] != attempt:
-        _task_journal = (attempt, DurableJournal(build_task_storage(context), 
clean_up_after_run=True))
+    context, attempt = found
+    if (journal := _find_attempt_journal(attempt)) is not None:
+        return journal
+    _task_journal = (attempt, DurableJournal(build_task_storage(context), 
clean_up_after_run=True))
     return _task_journal[1]
diff --git 
a/providers/common/ai/src/airflow/providers/common/ai/fallback_exceptions.py 
b/providers/common/ai/src/airflow/providers/common/ai/fallback_exceptions.py
new file mode 100644
index 00000000000..a4f07c1b41d
--- /dev/null
+++ b/providers/common/ai/src/airflow/providers/common/ai/fallback_exceptions.py
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+The error a failed ``fallback_conn_ids`` chain raises.
+
+Kept apart from :mod:`airflow.providers.common.ai.exceptions`, which must stay 
importable without
+loading pydantic-ai: this class subclasses pydantic-ai's 
``FallbackExceptionGroup``.
+"""
+
+from __future__ import annotations
+
+from pydantic_ai.exceptions import FallbackExceptionGroup
+
+
+class FallbackChainExhaustedError(FallbackExceptionGroup):
+    """
+    Raised when every connection of a ``fallback_conn_ids`` chain failed.
+
+    Subclasses pydantic-ai's ``FallbackExceptionGroup``, so ``except`` clauses 
and
+    ``RetryRule(exception=FallbackExceptionGroup)`` written for it keep 
matching. The
+    operator raises it ``from`` the group the chain raised, which stays 
``__cause__``.
+
+    ``replayed_steps`` and ``recorded_steps`` are what the durable journal 
replayed and recorded
+    in this attempt before the chain failed, summed over every step kind, so a 
retry policy can
+    tell an attempt that advanced the run from one that failed where it 
started. They are
+    ``None`` when no durable journal was active.
+    """
+
+    replayed_steps: int | None = None
+    recorded_steps: int | None = None
+
+    @property
+    def member_errors(self) -> tuple[Exception, ...]:
+        """The error each connection raised, in chain order."""
+        return tuple(self.exceptions)
+
+    @classmethod
+    def from_group(
+        cls,
+        group: FallbackExceptionGroup,
+        *,
+        replayed_steps: int | None = None,
+        recorded_steps: int | None = None,
+    ) -> FallbackChainExhaustedError:
+        """Build the error from the group the chain raised, keeping its 
attempts and members."""
+        error = cls(group.message, list(group.exceptions))
+        # ``attempts`` arrived in pydantic-ai-slim 2.55.0; the provider still 
supports 2.46.0.
+        error.attempts = getattr(group, "attempts", ())
+        error.replayed_steps = replayed_steps
+        error.recorded_steps = recorded_steps
+        return error
diff --git 
a/providers/common/ai/src/airflow/providers/common/ai/mixins/cancellable_run.py 
b/providers/common/ai/src/airflow/providers/common/ai/mixins/cancellable_run.py
index 5e40ce85b50..623e924f845 100644
--- 
a/providers/common/ai/src/airflow/providers/common/ai/mixins/cancellable_run.py
+++ 
b/providers/common/ai/src/airflow/providers/common/ai/mixins/cancellable_run.py
@@ -22,6 +22,10 @@ import threading
 from typing import TYPE_CHECKING, Any
 
 from pydantic_ai import CancellationToken
+from pydantic_ai.exceptions import FallbackExceptionGroup
+
+from airflow.providers.common.ai.durable.journal import find_active_journal
+from airflow.providers.common.ai.fallback_exceptions import 
FallbackChainExhaustedError
 
 if TYPE_CHECKING:
     from pydantic_ai import Agent, AgentRunResult
@@ -50,10 +54,25 @@ class CancellableAgentRunMixin:
     def run_agent_sync(
         self, agent: Agent[Any, Any], user_prompt: Any, **run_kwargs: Any
     ) -> AgentRunResult[Any]:
-        """Call ``agent.run_sync`` under a fresh cancellation token held for 
:meth:`on_kill`."""
+        """
+        Call ``agent.run_sync`` under a fresh cancellation token held for 
:meth:`on_kill`.
+
+        A ``FallbackExceptionGroup`` from a failed chain is raised again as
+        
:class:`~airflow.providers.common.ai.fallback_exceptions.FallbackChainExhaustedError`,
 carrying
+        what the durable journal replayed and recorded so far.
+        """
         self._cancellation_token = CancellationToken()
         try:
             return agent.run_sync(user_prompt, 
cancellation_token=self._cancellation_token, **run_kwargs)
+        except FallbackExceptionGroup as group:
+            if isinstance(group, FallbackChainExhaustedError):
+                raise
+            journal = find_active_journal()
+            raise FallbackChainExhaustedError.from_group(
+                group,
+                replayed_steps=None if journal is None else 
journal.stats.replayed.total(),
+                recorded_steps=None if journal is None else 
journal.stats.recorded.total(),
+            ) from group
         finally:
             self._cancellation_token = None
 
diff --git 
a/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py 
b/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py
index 4f14db9dd0d..61843401f3a 100644
--- a/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py
+++ b/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py
@@ -45,7 +45,9 @@ from types import MappingProxyType
 from typing import TYPE_CHECKING, Any, ClassVar, Literal
 
 from pydantic import BaseModel
+from pydantic_ai.exceptions import FallbackExceptionGroup
 
+from airflow.providers.common.ai.fallback_exceptions import 
FallbackChainExhaustedError
 from airflow.providers.common.ai.policies.decision import check_bar
 from airflow.providers.common.ai.utils.decision import (
     BARE_OUTPUT_FIELD,
@@ -320,22 +322,32 @@ class _ModelRetryPolicy(RetryPolicy):
 
         return PydanticAIHook.get_hook(self.llm_conn_id, 
hook_params={"model_id": self.model_id})
 
-    def _prompt(self, exception: BaseException, try_number: int, max_tries: 
int) -> str:
-        if self.include_traceback:
-            text = "".join(traceback.format_exception(exception)).rstrip("\n")
-        else:
-            text = str(exception)
+    def _limit(self, text: str, *, keep_tail: bool = False) -> str:
         # Redact before truncating -- truncating first could cut a registered 
secret in half.
         if self.redactor is not None:
             text = self.redactor(text)
-        if len(text) > self.max_exception_length:
-            if self.include_traceback:
-                # Keep the tail: the innermost frames and the final exception 
line say the most.
-                text = f"(truncated) ...{text[-self.max_exception_length :]}"
-            else:
-                text = f"{text[: self.max_exception_length]}... (truncated)"
-        if not self.include_traceback:
-            text = f"{type(exception).__name__}: {text}"
+        if len(text) <= self.max_exception_length:
+            return text
+        if keep_tail:
+            # Keep the tail: the innermost frames and the final exception line 
say the most.
+            return f"(truncated) ...{text[-self.max_exception_length :]}"
+        return f"{text[: self.max_exception_length]}... (truncated)"
+
+    def _prompt(self, exception: BaseException, try_number: int, max_tries: 
int) -> str:
+        if self.include_traceback:
+            text = 
self._limit("".join(traceback.format_exception(exception)).rstrip("\n"), 
keep_tail=True)
+        else:
+            text = f"{type(exception).__name__}: {self._limit(str(exception))}"
+            # Each member gets its own limit: one long error must not push the 
last one, which
+            # decides ``fallback_rules``, out of the prompt.
+            member_errors = self._get_member_errors(exception)
+            for position, member in enumerate(member_errors, start=1):
+                text += (
+                    f"\nchain member {position} of {len(member_errors)}: "
+                    f"{type(member).__name__}: {self._limit(str(member))}"
+                )
+        if (progress := self._describe_journal_progress(exception)) is not 
None:
+            text += f"\n{progress}"
         # ``max_tries`` is the last try number that may still retry (core adds 
to it on a clear),
         # so the final attempt is ``max_tries + 1``, as core logs "Starting 
attempt N of M".
         return (
@@ -343,6 +355,21 @@ class _ModelRetryPolicy(RetryPolicy):
             f"(attempt {try_number} of {max_tries + 1}):\n\n{text}"
         )
 
+    @staticmethod
+    def _get_member_errors(exception: BaseException) -> tuple[BaseException, 
...]:
+        """Return the error each fallback connection raised, in chain order, 
or ``()`` for any other error."""
+        return tuple(exception.exceptions) if isinstance(exception, 
FallbackExceptionGroup) else ()
+
+    @staticmethod
+    def _describe_journal_progress(exception: BaseException) -> str | None:
+        """Say how far the failed attempt got on the durable journal, or None 
when it is not known."""
+        if not isinstance(exception, FallbackChainExhaustedError) or 
exception.replayed_steps is None:
+            return None
+        return (
+            f"Before the chain failed, this attempt replayed 
{exception.replayed_steps} and "
+            f"recorded {exception.recorded_steps} durable journal steps, so a 
retry resumes from there."
+        )
+
     def _run(
         self, agent: Agent[Any, Any], exception: BaseException, try_number: 
int, max_tries: int
     ) -> AgentRunResult[Any]:
@@ -356,11 +383,25 @@ class _ModelRetryPolicy(RetryPolicy):
     def _rules_decision(
         self, exception: BaseException, try_number: int, max_tries: int, 
context: Context | None
     ) -> RetryDecision:
-        """Return the rules floor's decision: a ``fallback_rules`` match, else 
the task's own retry behaviour."""
-        if self.fallback_rules:
-            return ExceptionRetryPolicy(rules=self.fallback_rules).evaluate(
-                exception, try_number, max_tries, context
-            )
+        """
+        Return the rules floor's decision: a ``fallback_rules`` match, else 
the task's own retry behaviour.
+
+        When every connection of a fallback chain failed, a rule matches the 
chain's own error or the
+        error of its last connection, whichever the rule names, in the order 
the rules are listed.
+        The last connection decides when the members disagree: a 400 there is 
not retried because
+        an earlier connection was rate limited, and the reverse. The reason of 
a decision taken on
+        the last connection's error says so.
+        """
+        member_errors = self._get_member_errors(exception)
+        targets = [exception, member_errors[-1]] if member_errors else 
[exception]
+        for rule in self.fallback_rules or ():
+            for target in targets:
+                decision = ExceptionRetryPolicy(rules=[rule]).evaluate(target, 
try_number, max_tries, context)
+                if decision.reason is None:
+                    continue
+                if target is exception:
+                    return decision
+                return replace(decision, reason=f"{decision.reason} (last 
chain member)")
         return RetryDecision.default()
 
 
diff --git a/providers/common/ai/tests/unit/common/ai/durable/test_journal.py 
b/providers/common/ai/tests/unit/common/ai/durable/test_journal.py
index e244033d6f1..6ef4e51a0ee 100644
--- a/providers/common/ai/tests/unit/common/ai/durable/test_journal.py
+++ b/providers/common/ai/tests/unit/common/ai/durable/test_journal.py
@@ -36,6 +36,7 @@ from airflow.providers.common.ai.durable.journal import (
     DurableJournal,
     DurableRun,
     current_journal,
+    find_active_journal,
     journal_scope,
 )
 
@@ -1647,3 +1648,44 @@ class TestCurrentJournal:
             ti.id = "ti-2"
 
             assert current_journal() is not first
+
+
+class TestFindActiveJournal:
+    def test_a_scoped_journal_is_returned(self, memory_storage):
+        journal = DurableJournal(memory_storage)
+
+        with journal_scope(journal):
+            assert find_active_journal() is journal
+        assert find_active_journal() is None
+
+    def test_inside_a_task_it_does_not_create_the_journal(self, 
memory_storage, monkeypatch):
+        monkeypatch.setattr(journal_module, "_task_journal", None)
+        context = {"task_instance": SimpleNamespace(id="ti-1")}
+        with (
+            mock.patch.object(journal_module, "get_current_context", 
autospec=True, return_value=context),
+            mock.patch.object(
+                journal_module, "build_task_storage", autospec=True, 
return_value=memory_storage
+            ) as build_storage,
+        ):
+            assert find_active_journal() is None
+
+        build_storage.assert_not_called()
+        assert journal_module._task_journal is None
+
+    def test_it_returns_the_journal_the_attempt_already_created(self, 
memory_storage, monkeypatch):
+        monkeypatch.setattr(journal_module, "_task_journal", None)
+        ti = SimpleNamespace(id="ti-1")
+        with (
+            mock.patch.object(
+                journal_module, "get_current_context", autospec=True, 
return_value={"task_instance": ti}
+            ),
+            mock.patch.object(
+                journal_module, "build_task_storage", autospec=True, 
return_value=memory_storage
+            ),
+        ):
+            created = current_journal()
+            assert find_active_journal() is created
+
+            ti.id = "ti-2"
+
+            assert find_active_journal() is None
diff --git 
a/providers/common/ai/tests/unit/common/ai/mixins/test_cancellable_run.py 
b/providers/common/ai/tests/unit/common/ai/mixins/test_cancellable_run.py
index 69cf184dbc8..afe89fceeb6 100644
--- a/providers/common/ai/tests/unit/common/ai/mixins/test_cancellable_run.py
+++ b/providers/common/ai/tests/unit/common/ai/mixins/test_cancellable_run.py
@@ -18,15 +18,34 @@ from __future__ import annotations
 
 import threading
 import time
+from types import SimpleNamespace
+from unittest import mock
 from unittest.mock import DEFAULT, MagicMock
 
 import pytest
-from pydantic_ai import CancellationToken
-
+from pydantic_ai import Agent, CancellationToken
+from pydantic_ai.exceptions import FallbackExceptionGroup, ModelHTTPError
+from pydantic_ai.messages import ModelResponse, ToolCallPart
+from pydantic_ai.models.fallback import FallbackModel
+from pydantic_ai.models.function import AgentInfo, FunctionModel
+from pydantic_ai.toolsets import FunctionToolset
+
+from airflow.providers.common.ai.durable import AirflowDurability, journal as 
journal_module
+from airflow.providers.common.ai.durable.journal import DurableJournal, 
journal_scope
+from airflow.providers.common.ai.fallback_exceptions import 
FallbackChainExhaustedError
 from airflow.providers.common.ai.mixins.cancellable_run import 
CancellableAgentRunMixin
 from airflow.providers.common.ai.operators.agent import AgentOperator
 from airflow.providers.common.ai.operators.llm import LLMOperator
 
+from unit.common.ai.durable.memory_storage import MemoryStorage
+
+
+def _failing_member(status_code: int):
+    def fail(messages, info: AgentInfo) -> ModelResponse:
+        raise ModelHTTPError(status_code=status_code, model_name="member")
+
+    return fail
+
 
 class TestRunAgentSync:
     def test_forwards_held_cancellation_token_and_clears_after_success(self):
@@ -63,6 +82,112 @@ class TestRunAgentSync:
         assert mixin._cancellation_token is None
 
 
+class TestFallbackChainExhausted:
+    @pytest.fixture
+    def task_context(self, monkeypatch):
+        monkeypatch.setattr(journal_module, "_task_journal", None)
+        context = {"task_instance": SimpleNamespace(id="ti-1")}
+        with (
+            mock.patch.object(journal_module, "get_current_context", 
autospec=True, return_value=context),
+            mock.patch.object(
+                journal_module, "build_task_storage", autospec=True, 
return_value=MemoryStorage()
+            ),
+        ):
+            yield
+
+    def 
test_group_is_raised_again_as_chain_error_with_members_in_chain_order(self):
+        # Without the ``except FallbackExceptionGroup`` in run_agent_sync the 
run raises the bare
+        # group, and ``pytest.raises(FallbackChainExhaustedError)`` fails.
+        chain = FallbackModel(FunctionModel(_failing_member(429)), 
FunctionModel(_failing_member(400)))
+
+        with pytest.raises(FallbackChainExhaustedError) as caught:
+            CancellableAgentRunMixin().run_agent_sync(Agent(chain), "go")
+
+        error = caught.value
+        assert isinstance(error, FallbackExceptionGroup)
+        assert isinstance(error.__cause__, FallbackExceptionGroup)
+        assert not isinstance(error.__cause__, FallbackChainExhaustedError)
+        assert error.member_errors == error.__cause__.exceptions
+        assert [member.status_code for member in error.member_errors] == [429, 
400]
+        assert error.attempts == getattr(error.__cause__, "attempts", ())
+        # Outside a task: no journal.
+        assert error.replayed_steps is None
+        assert error.recorded_steps is None
+
+    @pytest.mark.usefixtures("task_context")
+    def test_inside_a_task_without_a_journal_reports_no_progress(self):
+        chain = FallbackModel(FunctionModel(_failing_member(429)), 
FunctionModel(_failing_member(400)))
+
+        with pytest.raises(FallbackChainExhaustedError) as caught:
+            CancellableAgentRunMixin().run_agent_sync(Agent(chain), "go")
+
+        assert caught.value.replayed_steps is None
+        assert caught.value.recorded_steps is None
+
+    @pytest.mark.usefixtures("task_context")
+    def test_inside_a_task_reports_the_journal_an_attempt_already_kept(self):
+        journal = journal_module.current_journal()
+        assert journal is not None
+        journal.stats.recorded["model"] = 1
+        journal.stats.replayed["tool"] = 2
+        chain = FallbackModel(FunctionModel(_failing_member(429)), 
FunctionModel(_failing_member(400)))
+
+        with pytest.raises(FallbackChainExhaustedError) as caught:
+            CancellableAgentRunMixin().run_agent_sync(Agent(chain), "go")
+
+        assert (caught.value.replayed_steps, caught.value.recorded_steps) == 
(2, 1)
+
+    def 
test_durable_retry_reports_what_the_attempt_replayed_and_recorded(self):
+        # Drops ``replayed_steps``/``recorded_steps`` (hard-coded None) -> 
both asserts below fail.
+        storage = MemoryStorage()
+        mixin = CancellableAgentRunMixin()
+        failing_after = {"responses": 1}
+
+        def member(messages, info: AgentInfo) -> ModelResponse:
+            answered = sum(isinstance(message, ModelResponse) for message in 
messages)
+            if answered >= failing_after["responses"]:
+                raise ModelHTTPError(status_code=429, model_name="member")
+            return ModelResponse(parts=[ToolCallPart("query", {"sql": "select 
1"})])
+
+        def build() -> Agent[None, str]:
+            toolset = FunctionToolset(id="db")
+
+            @toolset.tool_plain
+            def query(sql: str) -> str:
+                return "3 rows"
+
+            return Agent(
+                FallbackModel(FunctionModel(member), FunctionModel(member)),
+                name="analyst",
+                toolsets=[toolset],
+                capabilities=[AirflowDurability()],
+            )
+
+        def run_attempt() -> FallbackChainExhaustedError:
+            with journal_scope(DurableJournal(storage)):
+                with pytest.raises(FallbackChainExhaustedError) as caught:
+                    mixin.run_agent_sync(build(), "go")
+            return caught.value
+
+        first = run_attempt()
+        # The second request of the first attempt fails; one model step and 
one tool step were recorded.
+        assert (first.replayed_steps, first.recorded_steps) == (0, 2)
+
+        failing_after["responses"] = 2
+        second = run_attempt()
+        assert (second.replayed_steps, second.recorded_steps) == (2, 2)
+
+    def test_error_that_is_already_a_chain_error_is_not_wrapped_again(self):
+        agent = MagicMock(spec=["run_sync"])
+        original = FallbackChainExhaustedError("all failed", 
[RuntimeError("a")])
+        agent.run_sync.side_effect = original
+
+        with pytest.raises(FallbackChainExhaustedError) as caught:
+            CancellableAgentRunMixin().run_agent_sync(agent, "go")
+
+        assert caught.value is original
+
+
 class TestOnKill:
     def test_noop_when_no_run_active(self):
         mixin = CancellableAgentRunMixin()
diff --git a/providers/common/ai/tests/unit/common/ai/policies/test_retry.py 
b/providers/common/ai/tests/unit/common/ai/policies/test_retry.py
index 33e638eeb4d..12c91b454c7 100644
--- a/providers/common/ai/tests/unit/common/ai/policies/test_retry.py
+++ b/providers/common/ai/tests/unit/common/ai/policies/test_retry.py
@@ -32,7 +32,7 @@ import pytest
 from pydantic import TypeAdapter, ValidationError
 from pydantic_ai import Agent
 from pydantic_ai.agent import AgentRunResult
-from pydantic_ai.exceptions import UnexpectedModelBehavior
+from pydantic_ai.exceptions import FallbackExceptionGroup, 
UnexpectedModelBehavior
 from pydantic_ai.messages import ModelResponse
 
 # LLMRetryPolicy depends on the RetryPolicy ABC introduced in Airflow 3.3 
(AIP-105).
@@ -40,6 +40,7 @@ from pydantic_ai.messages import ModelResponse
 pytest.importorskip("airflow.sdk.definitions.retry_policy", 
reason="RetryPolicy requires Airflow 3.3+")
 
 from airflow.models.connection import Connection
+from airflow.providers.common.ai.fallback_exceptions import 
FallbackChainExhaustedError
 from airflow.providers.common.ai.hooks.pydantic_ai import PydanticAIAzureHook, 
PydanticAIHook
 from airflow.providers.common.ai.policies import retry as retry_module
 from airflow.providers.common.ai.policies.retry import (
@@ -955,6 +956,160 @@ class TestPrompt:
         assert prompt == f"{PROMPT_HEADER}{expected}"
 
 
+class _RateLimited(Exception):
+    """A connection answered 429."""
+
+
+class _BadRequest(Exception):
+    """A connection answered 400."""
+
+
+def _exhausted_chain(*members: Exception, replayed=None, recorded=None) -> 
FallbackChainExhaustedError:
+    return FallbackChainExhaustedError.from_group(
+        FallbackExceptionGroup("All models from FallbackModel failed", 
list(members)),
+        replayed_steps=replayed,
+        recorded_steps=recorded,
+    )
+
+
+class TestFallbackChain:
+    """A failed chain reaches the policy as one error; the policy still sees 
the members and the progress."""
+
+    @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+    @patch(HOOK, autospec=True)
+    def test_prompt_names_every_member_error_in_chain_order(self, 
mock_hook_cls, policy_cls):
+        # Without the member lines in ``_prompt`` the model sees only the 
group's message.
+        error = _exhausted_chain(_RateLimited("primary throttled"), 
_BadRequest("secondary rejected"))
+
+        prompt = _prompt_for(mock_hook_cls, policy_cls, error)
+
+        assert prompt.index("chain member 1 of 2: _RateLimited: primary 
throttled") < prompt.index(
+            "chain member 2 of 2: _BadRequest: secondary rejected"
+        )
+
+    @pytest.mark.enable_redact
+    @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+    @patch(HOOK, autospec=True)
+    def test_prompt_redacts_a_secret_in_a_member_error(self, mock_hook_cls, 
policy_cls):
+        # Dropping the member lines from the redacted ``text`` leaks the 
secret into the prompt.
+        reset_secrets_masker()
+        secret_value = "member-secret-token"
+        mask_secret(secret_value)
+        error = _exhausted_chain(_BadRequest(f"bad token {secret_value}"), 
_RateLimited("slow down"))
+
+        prompt = _prompt_for(mock_hook_cls, policy_cls, error)
+
+        assert "chain member 1 of 2: _BadRequest: bad token ***" in prompt
+        assert secret_value not in prompt
+
+    @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+    @patch(HOOK, autospec=True)
+    def 
test_a_long_earlier_member_does_not_push_the_last_one_out_of_the_prompt(
+        self, mock_hook_cls, policy_cls
+    ):
+        # Truncating the joined text instead of each member keeps the head and 
drops the last member.
+        error = _exhausted_chain(_RateLimited("x" * 500), _BadRequest("final 
rejection"))
+
+        prompt = _prompt_for(mock_hook_cls, policy_cls, error, 
max_exception_length=100)
+
+        assert "chain member 1 of 2: _RateLimited: " + "x" * 100 + "... 
(truncated)" in prompt
+        assert "chain member 2 of 2: _BadRequest: final rejection" in prompt
+
+    @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+    @patch(HOOK, autospec=True)
+    def test_prompt_says_how_far_the_durable_attempt_got(self, mock_hook_cls, 
policy_cls):
+        error = _exhausted_chain(_RateLimited("a"), _RateLimited("b"), 
replayed=2, recorded=1)
+
+        prompt = _prompt_for(mock_hook_cls, policy_cls, error)
+
+        assert "replayed 2 and recorded 1 durable journal steps" in prompt
+
+    @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+    @patch(HOOK, autospec=True)
+    def test_prompt_has_no_progress_line_without_a_journal(self, 
mock_hook_cls, policy_cls):
+        error = _exhausted_chain(_RateLimited("a"), _RateLimited("b"))
+
+        prompt = _prompt_for(mock_hook_cls, policy_cls, error)
+
+        assert "journal" not in prompt
+
+    @patch(HOOK, autospec=True)
+    def test_traceback_prompt_is_left_to_the_exception_group_formatter(self, 
mock_hook_cls):
+        error = _exhausted_chain(_RateLimited("a"), _BadRequest("b"))
+
+        prompt = _prompt_for(mock_hook_cls, ClassifierRetryPolicy, error, 
include_traceback=True)
+
+        assert "chain member" not in prompt
+        assert "_BadRequest: b" in prompt
+
+    @pytest.fixture
+    def rules(self):
+        return [
+            RetryRule(exception=_RateLimited, action=RetryAction.RETRY, 
retry_delay=timedelta(seconds=60)),
+            RetryRule(exception=_BadRequest, action=RetryAction.FAIL, 
reason="bad request"),
+        ]
+
+    @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+    def test_rules_match_the_members_when_all_are_rate_limited(self, rules, 
policy_cls):
+        # Without the last-member target, only the group is compared and no 
rule matches.
+        policy = policy_cls(llm_conn_id="nonexistent", fallback_rules=rules)
+
+        decision = policy.evaluate(
+            _exhausted_chain(_RateLimited("a"), _RateLimited("b")), 
try_number=1, max_tries=3
+        )
+
+        assert (decision.action, decision.retry_delay) == (RetryAction.RETRY, 
timedelta(seconds=60))
+        assert decision.reason.endswith("(last chain member)")
+
+    @pytest.mark.parametrize(
+        ("members", "expected"),
+        [
+            pytest.param((_RateLimited("a"), _BadRequest("b")), 
RetryAction.FAIL, id="last-is-400"),
+            pytest.param((_BadRequest("a"), _RateLimited("b")), 
RetryAction.RETRY, id="last-is-429"),
+        ],
+    )
+    def test_the_last_member_decides_when_the_members_disagree(self, rules, 
members, expected):
+        policy = LLMRetryPolicy(llm_conn_id="nonexistent", 
fallback_rules=rules)
+
+        decision = policy.evaluate(_exhausted_chain(*members), try_number=1, 
max_tries=3)
+
+        assert decision.action == expected
+
+    def test_a_rule_on_the_chain_error_itself_keeps_matching(self):
+        policy = LLMRetryPolicy(
+            llm_conn_id="nonexistent",
+            fallback_rules=[RetryRule(exception=FallbackExceptionGroup, 
action=RetryAction.FAIL)],
+        )
+
+        decision = policy.evaluate(_exhausted_chain(_RateLimited("a")), 
try_number=1, max_tries=3)
+
+        assert decision.action == RetryAction.FAIL
+        assert not decision.reason.endswith("(last chain member)")
+
+    def 
test_rules_apply_in_the_order_they_are_listed_across_both_targets(self):
+        # The member rule listed first wins over a group catch-all listed 
after it. Going target by
+        # target would reach the group catch-all first and fail.
+        policy = LLMRetryPolicy(
+            llm_conn_id="nonexistent",
+            fallback_rules=[
+                RetryRule(exception=_RateLimited, action=RetryAction.RETRY),
+                RetryRule(exception=FallbackExceptionGroup, 
action=RetryAction.FAIL, reason="group"),
+            ],
+        )
+
+        decision = policy.evaluate(_exhausted_chain(_RateLimited("a")), 
try_number=1, max_tries=3)
+
+        assert decision.action == RetryAction.RETRY
+        assert decision.reason.endswith("(last chain member)")
+
+    def test_a_chain_with_no_matching_rule_keeps_the_task_default(self, rules):
+        policy = LLMRetryPolicy(llm_conn_id="nonexistent", 
fallback_rules=rules)
+
+        decision = policy.evaluate(_exhausted_chain(ValueError("a")), 
try_number=1, max_tries=3)
+
+        assert decision.action == RetryAction.DEFAULT
+
+
 class TestFallbackBehaviour:
     """When the LLM call itself fails the deterministic path decides, 
unchanged."""
 
diff --git 
a/providers/common/ai/tests/unit/common/ai/test_fallback_exceptions.py 
b/providers/common/ai/tests/unit/common/ai/test_fallback_exceptions.py
new file mode 100644
index 00000000000..a86fb335ae3
--- /dev/null
+++ b/providers/common/ai/tests/unit/common/ai/test_fallback_exceptions.py
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pydantic_ai.exceptions import FallbackExceptionGroup
+
+from airflow.providers.common.ai.fallback_exceptions import 
FallbackChainExhaustedError
+
+
+class TestFromGroup:
+    def test_keeps_message_members_and_journal_progress(self):
+        members = [ValueError("first"), RuntimeError("second")]
+        group = FallbackExceptionGroup("All models from FallbackModel failed", 
members)
+
+        error = FallbackChainExhaustedError.from_group(group, 
replayed_steps=2, recorded_steps=1)
+
+        assert error.message == group.message
+        assert error.member_errors == tuple(members)
+        assert error.replayed_steps == 2
+        assert error.recorded_steps == 1
+
+    def test_group_without_attempts_yields_empty_attempts(self):
+        # Stands in for the group pydantic-ai-slim raised before 2.55.0, which 
had no ``attempts``.
+        group = ExceptionGroup("failed", [ValueError("x")])
+
+        error = FallbackChainExhaustedError.from_group(group)  # type: 
ignore[arg-type]
+
+        assert error.attempts == ()

Reply via email to