This is an automated email from the ASF dual-hosted git repository.
Lee-W pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 5a0a26f90ea Show common.ai retry policies why each fallback connection
failed (#76999)
5a0a26f90ea is described below
commit 5a0a26f90ea24be037e5d5b779e46752dd9853d2
Author: Wei Lee <[email protected]>
AuthorDate: Wed Nov 18 22:12:16 2026 +0800
Show common.ai retry policies why each fallback connection failed (#76999)
---
providers/common/ai/docs/changelog.rst | 9 ++
providers/common/ai/docs/provider_fallback.rst | 26 +++-
providers/common/ai/docs/recovery.rst | 30 ++--
providers/common/ai/docs/retry_policies.rst | 41 ++++--
.../airflow/providers/common/ai/durable/journal.py | 49 +++++--
.../providers/common/ai/fallback_exceptions.py | 65 +++++++++
.../providers/common/ai/mixins/cancellable_run.py | 21 ++-
.../airflow/providers/common/ai/policies/retry.py | 77 +++++++---
.../tests/unit/common/ai/durable/test_journal.py | 42 ++++++
.../unit/common/ai/mixins/test_cancellable_run.py | 129 ++++++++++++++++-
.../ai/tests/unit/common/ai/policies/test_retry.py | 157 ++++++++++++++++++++-
.../unit/common/ai/test_fallback_exceptions.py | 42 ++++++
12 files changed, 634 insertions(+), 54 deletions(-)
diff --git a/providers/common/ai/docs/changelog.rst
b/providers/common/ai/docs/changelog.rst
index fab49897489..f3281ab5c1f 100644
--- a/providers/common/ai/docs/changelog.rst
+++ b/providers/common/ai/docs/changelog.rst
@@ -25,6 +25,15 @@
Changelog
---------
+.. note::
+ When every connection of a ``fallback_conn_ids`` chain fails,
``LLMOperator`` and ``AgentOperator``
+ now raise
``airflow.providers.common.ai.fallback_exceptions.FallbackChainExhaustedError``
instead of
+ pydantic-ai's ``FallbackExceptionGroup``. It subclasses
``FallbackExceptionGroup``, so ``except``
+ clauses and ``RetryRule(exception=FallbackExceptionGroup)`` keep matching,
and ``__cause__`` is the
+ group the chain raised. ``LLMRetryPolicy`` and ``ClassifierRetryPolicy`` now
also match their
+ ``fallback_rules`` against the last connection's error, and show the model
each connection's error
+ and how many durable journal steps the attempt replayed and recorded.
+
.. note::
``@task.llm``, ``@task.agent``, ``@task.llm_sql``, ``@task.llm_branch``,
``@task.llm_file_analysis`` and ``@task.llm_schema_compare`` now render only
the prompt returned
diff --git a/providers/common/ai/docs/provider_fallback.rst
b/providers/common/ai/docs/provider_fallback.rst
index 3dc2595f64a..11823b06b2e 100644
--- a/providers/common/ai/docs/provider_fallback.rst
+++ b/providers/common/ai/docs/provider_fallback.rst
@@ -140,11 +140,27 @@ vendor can answer the same prompt now; a bad prompt, an
exhausted quota on every
a permanent data error still has to be decided by the retry policy.
Adding a chain changes what the retry layer sees. When every connection in the
chain
-fails, the exception the task raises is
``pydantic_ai.exceptions.FallbackExceptionGroup``,
-not the last provider's own exception, so retry rules matched against a
provider-specific
-exception type stop matching. Before adding a chain to a connection that Dags
already use,
-read :doc:`retry_policies` -- the section "When the connection also carries a
fallback
-chain" spells out what to check.
+fails, the task raises a ``pydantic_ai.exceptions.FallbackExceptionGroup``,
not the last
+provider's own exception. Which one depends on how the agent runs:
+
+* ``LLMOperator``, ``AgentOperator``, their subclasses, and the matching
``@task.llm`` and
+ ``@task.agent`` decorators raise
+
:class:`~airflow.providers.common.ai.fallback_exceptions.FallbackChainExhaustedError`,
a
+ subclass of ``FallbackExceptionGroup``. The error each connection raised is
in its
+ ``member_errors``, and with ``durable=True`` its ``replayed_steps`` and
``recorded_steps``
+ say how far the attempt got.
+* In a plain ``@task`` that builds an agent with ``PydanticAIHook`` and runs
it, the task
+ raises pydantic-ai's own ``FallbackExceptionGroup``. It has no
``member_errors``,
+ ``replayed_steps`` or ``recorded_steps``; read each connection's error from
the group's
+ ``.exceptions``. An ``except FallbackChainExhaustedError`` there never
matches.
+
+On both paths ``LLMRetryPolicy`` and ``ClassifierRetryPolicy`` match their
+``fallback_rules`` against the chain's error and then against the last
connection's error,
+so a rule written for a provider-specific exception type keeps matching. A
plain
+``ExceptionRetryPolicy``, or any other policy that compares one exception
type, sees only
+the group, and a rule written for a provider-specific exception type stops
matching there.
+Before adding a chain to a connection that Dags already use, read
:doc:`retry_policies` --
+the section "When the connection also carries a fallback chain" spells out
what to check.
Costs to know before configuring a long chain
---------------------------------------------
diff --git a/providers/common/ai/docs/recovery.rst
b/providers/common/ai/docs/recovery.rst
index 9e0c83bd85f..5ce975973ec 100644
--- a/providers/common/ai/docs/recovery.rst
+++ b/providers/common/ai/docs/recovery.rst
@@ -42,7 +42,8 @@ What each failure costs
- Kept. The next connection receives the run's message history, including
every
tool result, and continues the run.
* - Every connection in the chain fails, and the task raises
- ``pydantic_ai.exceptions.FallbackExceptionGroup``
+ ``FallbackChainExhaustedError`` (a
``pydantic_ai.exceptions.FallbackExceptionGroup``;
+ see :doc:`provider_fallback` for which tasks raise it)
- Airflow ``retries`` with ``durable=True``
- Replayed. The retry replays the steps an earlier attempt recorded,
whichever
connection answered them, and continues on the first connection that
answers now.
@@ -125,10 +126,12 @@ Retries after the whole chain fails
-----------------------------------
With ``durable=True``, each step is recorded in the task instance's journal
whichever
-connection answered it. When every connection fails, the task raises
-``FallbackExceptionGroup`` and Airflow retries it. The retry replays the
recorded steps,
-then sends its first live request through the chain from the top. The run
continues on
-the first connection that answers, whether or not that is the one that
answered before.
+connection answered it. When every connection fails, the operator (or
``@task.llm`` /
+``@task.agent``) raises ``FallbackChainExhaustedError``, and a plain ``@task``
with
+``AirflowDurability`` raises the bare ``FallbackExceptionGroup`` (see
+:doc:`provider_fallback`). Airflow retries either one. The retry replays the
recorded steps,
+then sends its first live request through the chain from the top. The run
continues on the
+first connection that answers, whether or not that is the one that answered
before.
A recorded model step replays only if the retry's request has the same
fingerprint, and
for a chain the model in the fingerprint is the whole chain, whichever
connection
@@ -144,11 +147,18 @@ What is not covered
whose worker died before its result was recorded. Make such tools
idempotent; see
"Side effects and idempotency" in :doc:`durable_execution`.
- A request every provider rejects, such as a malformed one that each returns
a 400
- for, walks the whole chain on every attempt. A ``RetryRule`` cannot stop
those retries
- without stopping outage retries too: once a chain is configured, both raise
- ``FallbackExceptionGroup``, and a rule compares only that top-level type.
Retry
- policies need Airflow 3.3 or later; see "When the connection also carries a
fallback
- chain" in :doc:`retry_policies`.
+ for, walks the whole chain on every attempt. The error the task raises is a
+ ``FallbackExceptionGroup``: ``FallbackChainExhaustedError``, carrying each
connection's
+ error in ``member_errors``, from the operators and decorators, and the plain
group from a
+ hand-built agent (see :doc:`provider_fallback`). ``LLMRetryPolicy`` and
+ ``ClassifierRetryPolicy`` match their ``fallback_rules`` against the last
connection's
+ error too, so a rule can fail a permanent error such as a bad request
without failing
+ outage retries, provided the two raise different exception types. Telling a
rate limit
+ from a bad request needs ``ModelRateLimitError`` and
``ModelRequestRejectedError``
+ rules; see :doc:`retry_policies` for the pydantic-ai-slim versions that have
them. A plain
+ ``ExceptionRetryPolicy`` compares only the group, and a retry still counts
against
+ ``retries``. Retry policies need Airflow 3.3 or later; see "When the
connection also
+ carries a fallback chain" in :doc:`retry_policies`.
- No connection type builds the ``AnthropicVertex`` client that Claude on
Vertex AI
needs, so a chain cannot include Claude on Vertex AI. A Vertex AI connection
reaches
Gemini.
diff --git a/providers/common/ai/docs/retry_policies.rst
b/providers/common/ai/docs/retry_policies.rst
index e8eeab44a46..b5567cfb8cb 100644
--- a/providers/common/ai/docs/retry_policies.rst
+++ b/providers/common/ai/docs/retry_policies.rst
@@ -379,15 +379,38 @@ retry behaviour with no change to the Dag. Two things
follow:
Separately, and regardless of this policy: if the connection **the task
itself** uses to call
the LLM (for example ``llm_conn_id`` on ``LLMOperator`` or ``AgentOperator``)
carries a fallback
chain, the exception the task raises once that chain is exhausted is
-``pydantic_ai.exceptions.FallbackExceptionGroup``, not the last provider's own
exception.
-``RetryRule`` matches with ``isinstance``, so a rule written as
-``RetryRule(exception=ModelHTTPError, ...)`` -- in ``fallback_rules`` here or
in a plain
-``ExceptionRetryPolicy`` -- stops matching. Match
``pydantic_ai.exceptions.FallbackExceptionGroup``
-explicitly as well; its only common ancestor with ``ModelAPIError`` is
``Exception``, too
-broad to write a rule against. The original per-model exceptions are still
available on
-``FallbackExceptionGroup.exceptions``, but ``RetryRule`` only compares the
top-level
-exception type, so a rule set that told 429s apart from 400s collapses into
one rule once
-the chain is in play.
+:class:`~airflow.providers.common.ai.fallback_exceptions.FallbackChainExhaustedError`.
It subclasses
+``pydantic_ai.exceptions.FallbackExceptionGroup``, so a rule or ``except``
clause written for the
+group keeps matching, and it is raised ``from`` the group the chain raised.
The error each
+connection raised is in ``member_errors``, in chain order. This holds for the
operators, their
+subclasses and the ``@task.llm`` and ``@task.agent`` decorators; a plain
``@task`` that runs an
+agent built with ``PydanticAIHook`` raises the bare
``FallbackExceptionGroup``, without
+``member_errors``, and the rules below apply to it the same way (see
:doc:`provider_fallback`).
+
+``RetryRule`` still compares one exception type, so the policies here read the
members for it:
+in ``fallback_rules``, each rule is tried against the chain's own error and
then against the
+error of the **last** connection, rule by rule in the order listed. A rule
written as
+``RetryRule(exception=ModelHTTPError, ...)`` therefore matches again, and a
rule set that tells
+failures apart by exception type, such as retrying a rate-limit error but
failing a bad-request
+error, keeps telling them apart. To separate those two, name
``ModelRateLimitError`` (from
+pydantic-ai-slim 2.59.0) and ``ModelRequestRejectedError`` (from 2.62.0) in
the rules. This
+provider supports pydantic-ai-slim from 2.46.0. Before 2.59.0 a 429 is also a
plain
+``ModelHTTPError``; from 2.59.0 to 2.61.x a rule can single out a rate limit
but not a
+rejected request. When the connections failed with different errors, the last
one decides;
+an earlier connection's rate limit does not make a final bad request
retryable, and the
+reverse. A decision taken on the last connection's error says so in its
reason. A rule
+cannot tell two failures of the same type apart, such as two
``ModelHTTPError`` with
+different status codes; that needs a classifier. Only these policies look at
the members: a
+plain ``ExceptionRetryPolicy`` still compares the chain's own error, so match
+``FallbackExceptionGroup`` there.
+
+The model that classifies sees the same detail. The prompt lists each
connection's error by
+type and message, redacted like the rest of the error and limited to
``max_exception_length``
+characters each, so a long message from one connection does not push the last
one out, and,
+when the attempt kept a durable journal, how many steps the failed attempt
replayed and
+recorded (``replayed_steps`` and ``recorded_steps`` on the error, which only
the operator
+path sets). A retry still counts as an attempt against ``retries``: a policy
can choose a
+longer delay or a different category from that detail, but cannot make a retry
free.
What the model can and cannot do
--------------------------------
diff --git
a/providers/common/ai/src/airflow/providers/common/ai/durable/journal.py
b/providers/common/ai/src/airflow/providers/common/ai/durable/journal.py
index a3d7ad13bb1..77ce4e22c89 100644
--- a/providers/common/ai/src/airflow/providers/common/ai/durable/journal.py
+++ b/providers/common/ai/src/airflow/providers/common/ai/durable/journal.py
@@ -1069,6 +1069,40 @@ def journal_scope(journal: DurableJournal) ->
Iterator[DurableJournal]:
_ACTIVE_JOURNAL.reset(token)
+def _get_task_context_and_attempt() -> tuple[Any, str] | None:
+ """Return the running task's context and attempt key, or ``None`` outside
a task."""
+ try:
+ context = get_current_context()
+ except (RuntimeError, AirflowException):
+ # Airflow 3 raises RuntimeError outside a task, Airflow 2
AirflowException.
+ return None
+ return context, make_task_instance_run_key(context["task_instance"])
+
+
+def _find_attempt_journal(attempt: str) -> DurableJournal | None:
+ """Return the cached journal if it belongs to ``attempt``, else
``None``."""
+ if _task_journal is not None and _task_journal[0] == attempt:
+ return _task_journal[1]
+ return None
+
+
+def find_active_journal() -> DurableJournal | None:
+ """
+ Return the journal already recording this run, without creating one.
+
+ A journal set with :func:`journal_scope` wins. Otherwise, inside a running
task, this
+ returns the task attempt's journal if something already created it.
``None`` means no
+ journal is recording: outside a task, or inside one where nothing has used
durable
+ execution yet.
+ """
+ if (journal := _ACTIVE_JOURNAL.get()) is not None:
+ return journal
+ if (found := _get_task_context_and_attempt()) is None:
+ return None
+ _, attempt = found
+ return _find_attempt_journal(attempt)
+
+
def current_journal() -> DurableJournal | None:
"""
Return the active journal, or ``None`` outside an Airflow task.
@@ -1076,18 +1110,17 @@ def current_journal() -> DurableJournal | None:
A journal set with :func:`journal_scope` wins. Otherwise, inside a running
task,
this returns the task attempt's own journal, created on first use, which
deletes the
steps of each run as soon as it succeeds. Outside a task there is nothing
to replay
- into, so durable execution is off and the agent runs as it would without
it.
+ into, so durable execution is off and the agent runs as it would without
it. To read
+ the journal without creating it, use :func:`find_active_journal`.
"""
global _task_journal
if (journal := _ACTIVE_JOURNAL.get()) is not None:
return journal
- try:
- context = get_current_context()
- except (RuntimeError, AirflowException):
- # Airflow 3 raises RuntimeError outside a task, Airflow 2
AirflowException.
+ if (found := _get_task_context_and_attempt()) is None:
return None
- attempt = make_task_instance_run_key(context["task_instance"])
- if _task_journal is None or _task_journal[0] != attempt:
- _task_journal = (attempt, DurableJournal(build_task_storage(context),
clean_up_after_run=True))
+ context, attempt = found
+ if (journal := _find_attempt_journal(attempt)) is not None:
+ return journal
+ _task_journal = (attempt, DurableJournal(build_task_storage(context),
clean_up_after_run=True))
return _task_journal[1]
diff --git
a/providers/common/ai/src/airflow/providers/common/ai/fallback_exceptions.py
b/providers/common/ai/src/airflow/providers/common/ai/fallback_exceptions.py
new file mode 100644
index 00000000000..a4f07c1b41d
--- /dev/null
+++ b/providers/common/ai/src/airflow/providers/common/ai/fallback_exceptions.py
@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+The error a failed ``fallback_conn_ids`` chain raises.
+
+Kept apart from :mod:`airflow.providers.common.ai.exceptions`, which must stay
importable without
+loading pydantic-ai: this class subclasses pydantic-ai's
``FallbackExceptionGroup``.
+"""
+
+from __future__ import annotations
+
+from pydantic_ai.exceptions import FallbackExceptionGroup
+
+
+class FallbackChainExhaustedError(FallbackExceptionGroup):
+ """
+ Raised when every connection of a ``fallback_conn_ids`` chain failed.
+
+ Subclasses pydantic-ai's ``FallbackExceptionGroup``, so ``except`` clauses
and
+ ``RetryRule(exception=FallbackExceptionGroup)`` written for it keep
matching. The
+ operator raises it ``from`` the group the chain raised, which stays
``__cause__``.
+
+ ``replayed_steps`` and ``recorded_steps`` are what the durable journal
replayed and recorded
+ in this attempt before the chain failed, summed over every step kind, so a
retry policy can
+ tell an attempt that advanced the run from one that failed where it
started. They are
+ ``None`` when no durable journal was active.
+ """
+
+ replayed_steps: int | None = None
+ recorded_steps: int | None = None
+
+ @property
+ def member_errors(self) -> tuple[Exception, ...]:
+ """The error each connection raised, in chain order."""
+ return tuple(self.exceptions)
+
+ @classmethod
+ def from_group(
+ cls,
+ group: FallbackExceptionGroup,
+ *,
+ replayed_steps: int | None = None,
+ recorded_steps: int | None = None,
+ ) -> FallbackChainExhaustedError:
+ """Build the error from the group the chain raised, keeping its
attempts and members."""
+ error = cls(group.message, list(group.exceptions))
+ # ``attempts`` arrived in pydantic-ai-slim 2.55.0; the provider still
supports 2.46.0.
+ error.attempts = getattr(group, "attempts", ())
+ error.replayed_steps = replayed_steps
+ error.recorded_steps = recorded_steps
+ return error
diff --git
a/providers/common/ai/src/airflow/providers/common/ai/mixins/cancellable_run.py
b/providers/common/ai/src/airflow/providers/common/ai/mixins/cancellable_run.py
index 5e40ce85b50..623e924f845 100644
---
a/providers/common/ai/src/airflow/providers/common/ai/mixins/cancellable_run.py
+++
b/providers/common/ai/src/airflow/providers/common/ai/mixins/cancellable_run.py
@@ -22,6 +22,10 @@ import threading
from typing import TYPE_CHECKING, Any
from pydantic_ai import CancellationToken
+from pydantic_ai.exceptions import FallbackExceptionGroup
+
+from airflow.providers.common.ai.durable.journal import find_active_journal
+from airflow.providers.common.ai.fallback_exceptions import
FallbackChainExhaustedError
if TYPE_CHECKING:
from pydantic_ai import Agent, AgentRunResult
@@ -50,10 +54,25 @@ class CancellableAgentRunMixin:
def run_agent_sync(
self, agent: Agent[Any, Any], user_prompt: Any, **run_kwargs: Any
) -> AgentRunResult[Any]:
- """Call ``agent.run_sync`` under a fresh cancellation token held for
:meth:`on_kill`."""
+ """
+ Call ``agent.run_sync`` under a fresh cancellation token held for
:meth:`on_kill`.
+
+ A ``FallbackExceptionGroup`` from a failed chain is raised again as
+
:class:`~airflow.providers.common.ai.fallback_exceptions.FallbackChainExhaustedError`,
carrying
+ what the durable journal replayed and recorded so far.
+ """
self._cancellation_token = CancellationToken()
try:
return agent.run_sync(user_prompt,
cancellation_token=self._cancellation_token, **run_kwargs)
+ except FallbackExceptionGroup as group:
+ if isinstance(group, FallbackChainExhaustedError):
+ raise
+ journal = find_active_journal()
+ raise FallbackChainExhaustedError.from_group(
+ group,
+ replayed_steps=None if journal is None else
journal.stats.replayed.total(),
+ recorded_steps=None if journal is None else
journal.stats.recorded.total(),
+ ) from group
finally:
self._cancellation_token = None
diff --git
a/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py
b/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py
index 4f14db9dd0d..61843401f3a 100644
--- a/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py
+++ b/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py
@@ -45,7 +45,9 @@ from types import MappingProxyType
from typing import TYPE_CHECKING, Any, ClassVar, Literal
from pydantic import BaseModel
+from pydantic_ai.exceptions import FallbackExceptionGroup
+from airflow.providers.common.ai.fallback_exceptions import
FallbackChainExhaustedError
from airflow.providers.common.ai.policies.decision import check_bar
from airflow.providers.common.ai.utils.decision import (
BARE_OUTPUT_FIELD,
@@ -320,22 +322,32 @@ class _ModelRetryPolicy(RetryPolicy):
return PydanticAIHook.get_hook(self.llm_conn_id,
hook_params={"model_id": self.model_id})
- def _prompt(self, exception: BaseException, try_number: int, max_tries:
int) -> str:
- if self.include_traceback:
- text = "".join(traceback.format_exception(exception)).rstrip("\n")
- else:
- text = str(exception)
+ def _limit(self, text: str, *, keep_tail: bool = False) -> str:
# Redact before truncating -- truncating first could cut a registered
secret in half.
if self.redactor is not None:
text = self.redactor(text)
- if len(text) > self.max_exception_length:
- if self.include_traceback:
- # Keep the tail: the innermost frames and the final exception
line say the most.
- text = f"(truncated) ...{text[-self.max_exception_length :]}"
- else:
- text = f"{text[: self.max_exception_length]}... (truncated)"
- if not self.include_traceback:
- text = f"{type(exception).__name__}: {text}"
+ if len(text) <= self.max_exception_length:
+ return text
+ if keep_tail:
+ # Keep the tail: the innermost frames and the final exception line
say the most.
+ return f"(truncated) ...{text[-self.max_exception_length :]}"
+ return f"{text[: self.max_exception_length]}... (truncated)"
+
+ def _prompt(self, exception: BaseException, try_number: int, max_tries:
int) -> str:
+ if self.include_traceback:
+ text =
self._limit("".join(traceback.format_exception(exception)).rstrip("\n"),
keep_tail=True)
+ else:
+ text = f"{type(exception).__name__}: {self._limit(str(exception))}"
+ # Each member gets its own limit: one long error must not push the
last one, which
+ # decides ``fallback_rules``, out of the prompt.
+ member_errors = self._get_member_errors(exception)
+ for position, member in enumerate(member_errors, start=1):
+ text += (
+ f"\nchain member {position} of {len(member_errors)}: "
+ f"{type(member).__name__}: {self._limit(str(member))}"
+ )
+ if (progress := self._describe_journal_progress(exception)) is not
None:
+ text += f"\n{progress}"
# ``max_tries`` is the last try number that may still retry (core adds
to it on a clear),
# so the final attempt is ``max_tries + 1``, as core logs "Starting
attempt N of M".
return (
@@ -343,6 +355,21 @@ class _ModelRetryPolicy(RetryPolicy):
f"(attempt {try_number} of {max_tries + 1}):\n\n{text}"
)
+ @staticmethod
+ def _get_member_errors(exception: BaseException) -> tuple[BaseException,
...]:
+ """Return the error each fallback connection raised, in chain order,
or ``()`` for any other error."""
+ return tuple(exception.exceptions) if isinstance(exception,
FallbackExceptionGroup) else ()
+
+ @staticmethod
+ def _describe_journal_progress(exception: BaseException) -> str | None:
+ """Say how far the failed attempt got on the durable journal, or None
when it is not known."""
+ if not isinstance(exception, FallbackChainExhaustedError) or
exception.replayed_steps is None:
+ return None
+ return (
+ f"Before the chain failed, this attempt replayed
{exception.replayed_steps} and "
+ f"recorded {exception.recorded_steps} durable journal steps, so a
retry resumes from there."
+ )
+
def _run(
self, agent: Agent[Any, Any], exception: BaseException, try_number:
int, max_tries: int
) -> AgentRunResult[Any]:
@@ -356,11 +383,25 @@ class _ModelRetryPolicy(RetryPolicy):
def _rules_decision(
self, exception: BaseException, try_number: int, max_tries: int,
context: Context | None
) -> RetryDecision:
- """Return the rules floor's decision: a ``fallback_rules`` match, else
the task's own retry behaviour."""
- if self.fallback_rules:
- return ExceptionRetryPolicy(rules=self.fallback_rules).evaluate(
- exception, try_number, max_tries, context
- )
+ """
+ Return the rules floor's decision: a ``fallback_rules`` match, else
the task's own retry behaviour.
+
+ When every connection of a fallback chain failed, a rule matches the
chain's own error or the
+ error of its last connection, whichever the rule names, in the order
the rules are listed.
+ The last connection decides when the members disagree: a 400 there is
not retried because
+ an earlier connection was rate limited, and the reverse. The reason of
a decision taken on
+ the last connection's error says so.
+ """
+ member_errors = self._get_member_errors(exception)
+ targets = [exception, member_errors[-1]] if member_errors else
[exception]
+ for rule in self.fallback_rules or ():
+ for target in targets:
+ decision = ExceptionRetryPolicy(rules=[rule]).evaluate(target,
try_number, max_tries, context)
+ if decision.reason is None:
+ continue
+ if target is exception:
+ return decision
+ return replace(decision, reason=f"{decision.reason} (last
chain member)")
return RetryDecision.default()
diff --git a/providers/common/ai/tests/unit/common/ai/durable/test_journal.py
b/providers/common/ai/tests/unit/common/ai/durable/test_journal.py
index e244033d6f1..6ef4e51a0ee 100644
--- a/providers/common/ai/tests/unit/common/ai/durable/test_journal.py
+++ b/providers/common/ai/tests/unit/common/ai/durable/test_journal.py
@@ -36,6 +36,7 @@ from airflow.providers.common.ai.durable.journal import (
DurableJournal,
DurableRun,
current_journal,
+ find_active_journal,
journal_scope,
)
@@ -1647,3 +1648,44 @@ class TestCurrentJournal:
ti.id = "ti-2"
assert current_journal() is not first
+
+
+class TestFindActiveJournal:
+ def test_a_scoped_journal_is_returned(self, memory_storage):
+ journal = DurableJournal(memory_storage)
+
+ with journal_scope(journal):
+ assert find_active_journal() is journal
+ assert find_active_journal() is None
+
+ def test_inside_a_task_it_does_not_create_the_journal(self,
memory_storage, monkeypatch):
+ monkeypatch.setattr(journal_module, "_task_journal", None)
+ context = {"task_instance": SimpleNamespace(id="ti-1")}
+ with (
+ mock.patch.object(journal_module, "get_current_context",
autospec=True, return_value=context),
+ mock.patch.object(
+ journal_module, "build_task_storage", autospec=True,
return_value=memory_storage
+ ) as build_storage,
+ ):
+ assert find_active_journal() is None
+
+ build_storage.assert_not_called()
+ assert journal_module._task_journal is None
+
+ def test_it_returns_the_journal_the_attempt_already_created(self,
memory_storage, monkeypatch):
+ monkeypatch.setattr(journal_module, "_task_journal", None)
+ ti = SimpleNamespace(id="ti-1")
+ with (
+ mock.patch.object(
+ journal_module, "get_current_context", autospec=True,
return_value={"task_instance": ti}
+ ),
+ mock.patch.object(
+ journal_module, "build_task_storage", autospec=True,
return_value=memory_storage
+ ),
+ ):
+ created = current_journal()
+ assert find_active_journal() is created
+
+ ti.id = "ti-2"
+
+ assert find_active_journal() is None
diff --git
a/providers/common/ai/tests/unit/common/ai/mixins/test_cancellable_run.py
b/providers/common/ai/tests/unit/common/ai/mixins/test_cancellable_run.py
index 69cf184dbc8..afe89fceeb6 100644
--- a/providers/common/ai/tests/unit/common/ai/mixins/test_cancellable_run.py
+++ b/providers/common/ai/tests/unit/common/ai/mixins/test_cancellable_run.py
@@ -18,15 +18,34 @@ from __future__ import annotations
import threading
import time
+from types import SimpleNamespace
+from unittest import mock
from unittest.mock import DEFAULT, MagicMock
import pytest
-from pydantic_ai import CancellationToken
-
+from pydantic_ai import Agent, CancellationToken
+from pydantic_ai.exceptions import FallbackExceptionGroup, ModelHTTPError
+from pydantic_ai.messages import ModelResponse, ToolCallPart
+from pydantic_ai.models.fallback import FallbackModel
+from pydantic_ai.models.function import AgentInfo, FunctionModel
+from pydantic_ai.toolsets import FunctionToolset
+
+from airflow.providers.common.ai.durable import AirflowDurability, journal as
journal_module
+from airflow.providers.common.ai.durable.journal import DurableJournal,
journal_scope
+from airflow.providers.common.ai.fallback_exceptions import
FallbackChainExhaustedError
from airflow.providers.common.ai.mixins.cancellable_run import
CancellableAgentRunMixin
from airflow.providers.common.ai.operators.agent import AgentOperator
from airflow.providers.common.ai.operators.llm import LLMOperator
+from unit.common.ai.durable.memory_storage import MemoryStorage
+
+
+def _failing_member(status_code: int):
+ def fail(messages, info: AgentInfo) -> ModelResponse:
+ raise ModelHTTPError(status_code=status_code, model_name="member")
+
+ return fail
+
class TestRunAgentSync:
def test_forwards_held_cancellation_token_and_clears_after_success(self):
@@ -63,6 +82,112 @@ class TestRunAgentSync:
assert mixin._cancellation_token is None
+class TestFallbackChainExhausted:
+ @pytest.fixture
+ def task_context(self, monkeypatch):
+ monkeypatch.setattr(journal_module, "_task_journal", None)
+ context = {"task_instance": SimpleNamespace(id="ti-1")}
+ with (
+ mock.patch.object(journal_module, "get_current_context",
autospec=True, return_value=context),
+ mock.patch.object(
+ journal_module, "build_task_storage", autospec=True,
return_value=MemoryStorage()
+ ),
+ ):
+ yield
+
+ def
test_group_is_raised_again_as_chain_error_with_members_in_chain_order(self):
+ # Without the ``except FallbackExceptionGroup`` in run_agent_sync the
run raises the bare
+ # group, and ``pytest.raises(FallbackChainExhaustedError)`` fails.
+ chain = FallbackModel(FunctionModel(_failing_member(429)),
FunctionModel(_failing_member(400)))
+
+ with pytest.raises(FallbackChainExhaustedError) as caught:
+ CancellableAgentRunMixin().run_agent_sync(Agent(chain), "go")
+
+ error = caught.value
+ assert isinstance(error, FallbackExceptionGroup)
+ assert isinstance(error.__cause__, FallbackExceptionGroup)
+ assert not isinstance(error.__cause__, FallbackChainExhaustedError)
+ assert error.member_errors == error.__cause__.exceptions
+ assert [member.status_code for member in error.member_errors] == [429,
400]
+ assert error.attempts == getattr(error.__cause__, "attempts", ())
+ # Outside a task: no journal.
+ assert error.replayed_steps is None
+ assert error.recorded_steps is None
+
+ @pytest.mark.usefixtures("task_context")
+ def test_inside_a_task_without_a_journal_reports_no_progress(self):
+ chain = FallbackModel(FunctionModel(_failing_member(429)),
FunctionModel(_failing_member(400)))
+
+ with pytest.raises(FallbackChainExhaustedError) as caught:
+ CancellableAgentRunMixin().run_agent_sync(Agent(chain), "go")
+
+ assert caught.value.replayed_steps is None
+ assert caught.value.recorded_steps is None
+
+ @pytest.mark.usefixtures("task_context")
+ def test_inside_a_task_reports_the_journal_an_attempt_already_kept(self):
+ journal = journal_module.current_journal()
+ assert journal is not None
+ journal.stats.recorded["model"] = 1
+ journal.stats.replayed["tool"] = 2
+ chain = FallbackModel(FunctionModel(_failing_member(429)),
FunctionModel(_failing_member(400)))
+
+ with pytest.raises(FallbackChainExhaustedError) as caught:
+ CancellableAgentRunMixin().run_agent_sync(Agent(chain), "go")
+
+ assert (caught.value.replayed_steps, caught.value.recorded_steps) ==
(2, 1)
+
+ def
test_durable_retry_reports_what_the_attempt_replayed_and_recorded(self):
+ # Drops ``replayed_steps``/``recorded_steps`` (hard-coded None) ->
both asserts below fail.
+ storage = MemoryStorage()
+ mixin = CancellableAgentRunMixin()
+ failing_after = {"responses": 1}
+
+ def member(messages, info: AgentInfo) -> ModelResponse:
+ answered = sum(isinstance(message, ModelResponse) for message in
messages)
+ if answered >= failing_after["responses"]:
+ raise ModelHTTPError(status_code=429, model_name="member")
+ return ModelResponse(parts=[ToolCallPart("query", {"sql": "select
1"})])
+
+ def build() -> Agent[None, str]:
+ toolset = FunctionToolset(id="db")
+
+ @toolset.tool_plain
+ def query(sql: str) -> str:
+ return "3 rows"
+
+ return Agent(
+ FallbackModel(FunctionModel(member), FunctionModel(member)),
+ name="analyst",
+ toolsets=[toolset],
+ capabilities=[AirflowDurability()],
+ )
+
+ def run_attempt() -> FallbackChainExhaustedError:
+ with journal_scope(DurableJournal(storage)):
+ with pytest.raises(FallbackChainExhaustedError) as caught:
+ mixin.run_agent_sync(build(), "go")
+ return caught.value
+
+ first = run_attempt()
+ # The second request of the first attempt fails; one model step and
one tool step were recorded.
+ assert (first.replayed_steps, first.recorded_steps) == (0, 2)
+
+ failing_after["responses"] = 2
+ second = run_attempt()
+ assert (second.replayed_steps, second.recorded_steps) == (2, 2)
+
+ def test_error_that_is_already_a_chain_error_is_not_wrapped_again(self):
+ agent = MagicMock(spec=["run_sync"])
+ original = FallbackChainExhaustedError("all failed",
[RuntimeError("a")])
+ agent.run_sync.side_effect = original
+
+ with pytest.raises(FallbackChainExhaustedError) as caught:
+ CancellableAgentRunMixin().run_agent_sync(agent, "go")
+
+ assert caught.value is original
+
+
class TestOnKill:
def test_noop_when_no_run_active(self):
mixin = CancellableAgentRunMixin()
diff --git a/providers/common/ai/tests/unit/common/ai/policies/test_retry.py
b/providers/common/ai/tests/unit/common/ai/policies/test_retry.py
index 33e638eeb4d..12c91b454c7 100644
--- a/providers/common/ai/tests/unit/common/ai/policies/test_retry.py
+++ b/providers/common/ai/tests/unit/common/ai/policies/test_retry.py
@@ -32,7 +32,7 @@ import pytest
from pydantic import TypeAdapter, ValidationError
from pydantic_ai import Agent
from pydantic_ai.agent import AgentRunResult
-from pydantic_ai.exceptions import UnexpectedModelBehavior
+from pydantic_ai.exceptions import FallbackExceptionGroup,
UnexpectedModelBehavior
from pydantic_ai.messages import ModelResponse
# LLMRetryPolicy depends on the RetryPolicy ABC introduced in Airflow 3.3
(AIP-105).
@@ -40,6 +40,7 @@ from pydantic_ai.messages import ModelResponse
pytest.importorskip("airflow.sdk.definitions.retry_policy",
reason="RetryPolicy requires Airflow 3.3+")
from airflow.models.connection import Connection
+from airflow.providers.common.ai.fallback_exceptions import
FallbackChainExhaustedError
from airflow.providers.common.ai.hooks.pydantic_ai import PydanticAIAzureHook,
PydanticAIHook
from airflow.providers.common.ai.policies import retry as retry_module
from airflow.providers.common.ai.policies.retry import (
@@ -955,6 +956,160 @@ class TestPrompt:
assert prompt == f"{PROMPT_HEADER}{expected}"
+class _RateLimited(Exception):
+ """A connection answered 429."""
+
+
+class _BadRequest(Exception):
+ """A connection answered 400."""
+
+
+def _exhausted_chain(*members: Exception, replayed=None, recorded=None) ->
FallbackChainExhaustedError:
+ return FallbackChainExhaustedError.from_group(
+ FallbackExceptionGroup("All models from FallbackModel failed",
list(members)),
+ replayed_steps=replayed,
+ recorded_steps=recorded,
+ )
+
+
+class TestFallbackChain:
+ """A failed chain reaches the policy as one error; the policy still sees
the members and the progress."""
+
+ @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+ @patch(HOOK, autospec=True)
+ def test_prompt_names_every_member_error_in_chain_order(self,
mock_hook_cls, policy_cls):
+ # Without the member lines in ``_prompt`` the model sees only the
group's message.
+ error = _exhausted_chain(_RateLimited("primary throttled"),
_BadRequest("secondary rejected"))
+
+ prompt = _prompt_for(mock_hook_cls, policy_cls, error)
+
+ assert prompt.index("chain member 1 of 2: _RateLimited: primary
throttled") < prompt.index(
+ "chain member 2 of 2: _BadRequest: secondary rejected"
+ )
+
+ @pytest.mark.enable_redact
+ @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+ @patch(HOOK, autospec=True)
+ def test_prompt_redacts_a_secret_in_a_member_error(self, mock_hook_cls,
policy_cls):
+ # Dropping the member lines from the redacted ``text`` leaks the
secret into the prompt.
+ reset_secrets_masker()
+ secret_value = "member-secret-token"
+ mask_secret(secret_value)
+ error = _exhausted_chain(_BadRequest(f"bad token {secret_value}"),
_RateLimited("slow down"))
+
+ prompt = _prompt_for(mock_hook_cls, policy_cls, error)
+
+ assert "chain member 1 of 2: _BadRequest: bad token ***" in prompt
+ assert secret_value not in prompt
+
+ @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+ @patch(HOOK, autospec=True)
+ def
test_a_long_earlier_member_does_not_push_the_last_one_out_of_the_prompt(
+ self, mock_hook_cls, policy_cls
+ ):
+ # Truncating the joined text instead of each member keeps the head and
drops the last member.
+ error = _exhausted_chain(_RateLimited("x" * 500), _BadRequest("final
rejection"))
+
+ prompt = _prompt_for(mock_hook_cls, policy_cls, error,
max_exception_length=100)
+
+ assert "chain member 1 of 2: _RateLimited: " + "x" * 100 + "...
(truncated)" in prompt
+ assert "chain member 2 of 2: _BadRequest: final rejection" in prompt
+
+ @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+ @patch(HOOK, autospec=True)
+ def test_prompt_says_how_far_the_durable_attempt_got(self, mock_hook_cls,
policy_cls):
+ error = _exhausted_chain(_RateLimited("a"), _RateLimited("b"),
replayed=2, recorded=1)
+
+ prompt = _prompt_for(mock_hook_cls, policy_cls, error)
+
+ assert "replayed 2 and recorded 1 durable journal steps" in prompt
+
+ @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+ @patch(HOOK, autospec=True)
+ def test_prompt_has_no_progress_line_without_a_journal(self,
mock_hook_cls, policy_cls):
+ error = _exhausted_chain(_RateLimited("a"), _RateLimited("b"))
+
+ prompt = _prompt_for(mock_hook_cls, policy_cls, error)
+
+ assert "journal" not in prompt
+
+ @patch(HOOK, autospec=True)
+ def test_traceback_prompt_is_left_to_the_exception_group_formatter(self,
mock_hook_cls):
+ error = _exhausted_chain(_RateLimited("a"), _BadRequest("b"))
+
+ prompt = _prompt_for(mock_hook_cls, ClassifierRetryPolicy, error,
include_traceback=True)
+
+ assert "chain member" not in prompt
+ assert "_BadRequest: b" in prompt
+
+ @pytest.fixture
+ def rules(self):
+ return [
+ RetryRule(exception=_RateLimited, action=RetryAction.RETRY,
retry_delay=timedelta(seconds=60)),
+ RetryRule(exception=_BadRequest, action=RetryAction.FAIL,
reason="bad request"),
+ ]
+
+ @pytest.mark.parametrize("policy_cls", POLICY_CLASSES)
+ def test_rules_match_the_members_when_all_are_rate_limited(self, rules,
policy_cls):
+ # Without the last-member target, only the group is compared and no
rule matches.
+ policy = policy_cls(llm_conn_id="nonexistent", fallback_rules=rules)
+
+ decision = policy.evaluate(
+ _exhausted_chain(_RateLimited("a"), _RateLimited("b")),
try_number=1, max_tries=3
+ )
+
+ assert (decision.action, decision.retry_delay) == (RetryAction.RETRY,
timedelta(seconds=60))
+ assert decision.reason.endswith("(last chain member)")
+
+ @pytest.mark.parametrize(
+ ("members", "expected"),
+ [
+ pytest.param((_RateLimited("a"), _BadRequest("b")),
RetryAction.FAIL, id="last-is-400"),
+ pytest.param((_BadRequest("a"), _RateLimited("b")),
RetryAction.RETRY, id="last-is-429"),
+ ],
+ )
+ def test_the_last_member_decides_when_the_members_disagree(self, rules,
members, expected):
+ policy = LLMRetryPolicy(llm_conn_id="nonexistent",
fallback_rules=rules)
+
+ decision = policy.evaluate(_exhausted_chain(*members), try_number=1,
max_tries=3)
+
+ assert decision.action == expected
+
+ def test_a_rule_on_the_chain_error_itself_keeps_matching(self):
+ policy = LLMRetryPolicy(
+ llm_conn_id="nonexistent",
+ fallback_rules=[RetryRule(exception=FallbackExceptionGroup,
action=RetryAction.FAIL)],
+ )
+
+ decision = policy.evaluate(_exhausted_chain(_RateLimited("a")),
try_number=1, max_tries=3)
+
+ assert decision.action == RetryAction.FAIL
+ assert not decision.reason.endswith("(last chain member)")
+
+ def
test_rules_apply_in_the_order_they_are_listed_across_both_targets(self):
+ # The member rule listed first wins over a group catch-all listed
after it. Going target by
+ # target would reach the group catch-all first and fail.
+ policy = LLMRetryPolicy(
+ llm_conn_id="nonexistent",
+ fallback_rules=[
+ RetryRule(exception=_RateLimited, action=RetryAction.RETRY),
+ RetryRule(exception=FallbackExceptionGroup,
action=RetryAction.FAIL, reason="group"),
+ ],
+ )
+
+ decision = policy.evaluate(_exhausted_chain(_RateLimited("a")),
try_number=1, max_tries=3)
+
+ assert decision.action == RetryAction.RETRY
+ assert decision.reason.endswith("(last chain member)")
+
+ def test_a_chain_with_no_matching_rule_keeps_the_task_default(self, rules):
+ policy = LLMRetryPolicy(llm_conn_id="nonexistent",
fallback_rules=rules)
+
+ decision = policy.evaluate(_exhausted_chain(ValueError("a")),
try_number=1, max_tries=3)
+
+ assert decision.action == RetryAction.DEFAULT
+
+
class TestFallbackBehaviour:
"""When the LLM call itself fails the deterministic path decides,
unchanged."""
diff --git
a/providers/common/ai/tests/unit/common/ai/test_fallback_exceptions.py
b/providers/common/ai/tests/unit/common/ai/test_fallback_exceptions.py
new file mode 100644
index 00000000000..a86fb335ae3
--- /dev/null
+++ b/providers/common/ai/tests/unit/common/ai/test_fallback_exceptions.py
@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from pydantic_ai.exceptions import FallbackExceptionGroup
+
+from airflow.providers.common.ai.fallback_exceptions import
FallbackChainExhaustedError
+
+
+class TestFromGroup:
+ def test_keeps_message_members_and_journal_progress(self):
+ members = [ValueError("first"), RuntimeError("second")]
+ group = FallbackExceptionGroup("All models from FallbackModel failed",
members)
+
+ error = FallbackChainExhaustedError.from_group(group,
replayed_steps=2, recorded_steps=1)
+
+ assert error.message == group.message
+ assert error.member_errors == tuple(members)
+ assert error.replayed_steps == 2
+ assert error.recorded_steps == 1
+
+ def test_group_without_attempts_yields_empty_attempts(self):
+ # Stands in for the group pydantic-ai-slim raised before 2.55.0, which
had no ``attempts``.
+ group = ExceptionGroup("failed", [ValueError("x")])
+
+ error = FallbackChainExhaustedError.from_group(group) # type:
ignore[arg-type]
+
+ assert error.attempts == ()