Lee-W commented on code in PR #72156: URL: https://github.com/apache/airflow/pull/72156#discussion_r4045851171
########## providers/common/ai/docs/provider_fallback.rst: ########## @@ -0,0 +1,208 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Provider fallback +================= + +A single ``llm_conn_id`` gives a task one provider. When that provider is down, the task +fails and retries into the same outage. ``fallback_conn_ids`` gives the connection an +ordered list of other connections to try, so a provider outage moves to the next vendor +inside the same task attempt. + +Configure it on the connection +------------------------------ + +Put the chain in the primary connection's extra: + +.. code-block:: json + + { + "model": "openai:gpt-5", + "fallback_conn_ids": ["anthropic_prod", "bedrock_dr"] + } + +Every entry is an Airflow connection ID, resolved through the hook registered for its own +connection type. A chain can therefore mix vendors whose credentials live in different +connection fields — ``pydanticai`` for OpenAI, ``pydanticai_bedrock`` for a Bedrock +standby — without the Dag knowing anything about either. + +That is the point of configuring it here rather than in Dag code: the Dag keeps naming one +connection, and whoever administers the connections owns the failover topology. Changing a +standby provider is a connection edit, not a Dag deployment. + +A *bare* model name (e.g. ``"gpt-5"`` rather than ``"openai:gpt-5"``) is forwarded down +the chain as a logical model name: each connection that has no ``model`` of its own +resolves that name against its own platform, so one bare name can reach a primary and +every fallback without repeating it per connection. It does not matter where the primary's +name comes from -- the ``Model`` field on its connection and a ``model_id`` on the operator +or hook are forwarded alike. A fallback with its own ``model`` in +extra always uses that instead -- this is how a fallback pins a spelling the forwarded +name would not produce, such as Bedrock's region-prefixed ``us.anthropic.`` model ids. A +name that already pins a platform (its segment before the first ``:`` is itself a +recognized provider, e.g. ``"openai:gpt-5"``) is *not* forwarded; a fallback with no +``model`` of its own still raises "no model specified" rather than trying a prefixed name +meant for a different provider. See :doc:`connections/pydantic_ai_azure`, +:doc:`connections/pydantic_ai_bedrock` and :doc:`connections/pydantic_ai_vertex` for how +each vendor connection resolves a bare name. + +Configure it on the operator +----------------------------- + +``fallback_conn_ids`` is also a parameter on +:class:`~airflow.providers.common.ai.operators.llm.LLMOperator`, +:class:`~airflow.providers.common.ai.operators.agent.AgentOperator`, their subclasses, +and the matching ``@task.llm`` / ``@task.agent`` decorators -- mirroring ``model_id``, +which is settable at the same two layers: + +.. exampleinclude:: /../../ai/src/airflow/providers/common/ai/example_dags/example_llm_fallback.py + :language: python + :dedent: 0 + :start-after: [START howto_llm_fallback_operator_argument] + :end-before: [END howto_llm_fallback_operator_argument] + +The operator argument overrides the connection's extra field, and passing ``[]`` +explicitly disables a chain configured there -- ``None`` (the default) reads whatever +the connection says. Use this when a task should own its own failover order instead of +inheriting it from however the connection is configured. + +Configure it in code +-------------------- + +:class:`~airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook` also takes the list +directly, which is what a task that constructs the hook itself (rather than through an +operator) should use: + +.. exampleinclude:: /../../ai/src/airflow/providers/common/ai/example_dags/example_llm_fallback.py + :language: python + :dedent: 0 + :start-after: [START howto_llm_fallback_hook_argument] + :end-before: [END howto_llm_fallback_hook_argument] + +The argument wins over the connection's extra, and passing ``[]`` explicitly disables a +chain configured there. Omitting it entirely (``None``) means "use whatever the connection +says", which is why the two are not interchangeable. + +Where this sits among the retry layers +-------------------------------------- + +Three mechanisms handle failure at different time scales, and they compose rather than +replace each other: + +.. list-table:: + :header-rows: 1 + :widths: 25 40 35 + + * - Scope + - Mechanism + - Handles + * - Within one model call + - ``fallback_conn_ids`` + - This vendor's API is returning errors; ask the next one (any ``ModelAPIError``, + transient or not) + * - Within one task attempt + - ``timeout`` in pydantic-ai's ``ModelSettings`` + - This vendor is slow rather than down + * - Across task attempts + - :doc:`retry_policies` (including ``LLMRetryPolicy``) + - Whether this failure is worth retrying at all + +A chain does not remove the need for the outer layers. It covers the case where another +vendor can answer the same prompt now; a bad prompt, an exhausted quota on every vendor, or +a permanent data error still has to be decided by the retry policy. + +Adding a chain changes what the retry layer sees. When every connection in the chain +fails, the exception the task raises is ``pydantic_ai.exceptions.FallbackExceptionGroup``, +not the last provider's own exception, so retry rules matched against a provider-specific +exception type stop matching. Before adding a chain to a connection that Dags already use, +read :doc:`retry_policies` -- the section "When the connection also carries a fallback +chain" spells out what to check. + +Costs to know before configuring a long chain +--------------------------------------------- + +**The timeout multiplies.** pydantic-ai applies a ``ModelSettings`` timeout to each model +in the chain, not to the chain as a whole. A 30-second timeout across three connections is +a 90-second worst case for one call. + +**There is no circuit breaker.** Every call tries the primary first. During an outage each +task instance pays the primary's timeout again before failing over, so 500 mapped tasks pay +it 500 times. Keeping the primary's timeout short bounds both of these. + +**Chains are not resolved recursively.** If a connection listed as a fallback declares its +own ``fallback_conn_ids``, resolution fails with an error rather than following it. List +every provider directly on the primary; a flat chain is the one you can read off a single +connection. + +**Non-transient errors still walk the whole chain.** Failover triggers on pydantic-ai's +``ModelAPIError`` family, which includes ``ModelHTTPError`` -- raised for any 4xx as well as +5xx. A malformed prompt, an expired key, or a misspelled model name is therefore retried Review Comment: Rewritten. Worth flagging that the gate from the cross-platform thread changes one of these: a bare name carrying a colon is no longer forwarded to a fallback on another platform, so a misspelled native id is not shared across a mixed-platform chain either. The section is about the malformed prompt now, which is the case that genuinely is shared. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
