This is an automated email from the ASF dual-hosted git repository.
Lee-W pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 6ea3de07624 Add feature-comparison table and toolset links to
common.ai provider docs (#69649)
6ea3de07624 is described below
commit 6ea3de07624762a037364c3ea3a4e13a1d8c3f3e
Author: Wei Lee <[email protected]>
AuthorDate: Fri Jul 10 09:33:37 2026 +0800
Add feature-comparison table and toolset links to common.ai provider docs
(#69649)
---
providers/anthropic/docs/index.rst | 6 +++++-
providers/cohere/docs/index.rst | 17 +++++++++++++++
providers/common/ai/docs/index.rst | 42 ++++++++++++++++++++++++++++++--------
providers/openai/docs/index.rst | 11 +++++++---
4 files changed, 64 insertions(+), 12 deletions(-)
diff --git a/providers/anthropic/docs/index.rst
b/providers/anthropic/docs/index.rst
index 3794d1ae58b..b6840352745 100644
--- a/providers/anthropic/docs/index.rst
+++ b/providers/anthropic/docs/index.rst
@@ -19,6 +19,9 @@
``apache-airflow-providers-anthropic``
======================================
+The ``anthropic`` provider gives Dags direct access to Anthropic's own APIs —
this page
+compares that choice against ``common.ai``.
+
When to use this provider
--------------------------
@@ -27,7 +30,8 @@ for you, which no vendor-neutral operator wraps:
* ``AnthropicBatchOperator`` and ``AnthropicBatchSensor`` — submit a Claude
`Message Batches
<https://docs.claude.com/en/docs/build-with-claude/batch-processing>`__
- job for asynchronous bulk processing and wait for it to complete.
+ job for asynchronous bulk processing and wait for it to complete; Message
Batches run at
+ 50% of standard cost, with most completing within an hour and a 24-hour SLA.
* ``AnthropicAgentSessionOperator`` — start a Managed Agents session in which
the agent loop
runs server-side on Anthropic's infrastructure; the Airflow task only kicks
off the session
and waits for its outcome.
diff --git a/providers/cohere/docs/index.rst b/providers/cohere/docs/index.rst
index 052babe4d9e..092912d34ef 100644
--- a/providers/cohere/docs/index.rst
+++ b/providers/cohere/docs/index.rst
@@ -19,6 +19,23 @@
``apache-airflow-providers-cohere``
======================================
+The ``cohere`` provider gives Dags direct access to Cohere's own Embed API —
this page
+compares that choice against ``common.ai``.
+
+When to use this provider
+--------------------------
+
+Use ``cohere`` when a Dag needs Cohere's native embedding models specifically:
+
+* ``CohereEmbeddingOperator`` — call Cohere's
+ `Embed API <https://docs.cohere.com/docs/embeddings>`__ directly via
``CohereHook``.
+
+Use :doc:`apache-airflow-providers-common-ai:index` instead when the embedding
step should
+stay vendor-neutral:
+
+* Document-to-vector-store pipelines with its document loader, embedding, and
retrieval
+ operators (see :doc:`apache-airflow-providers-common-ai:operators/index`),
which are not
+ tied to Cohere's embedding models.
.. toctree::
:hidden:
diff --git a/providers/common/ai/docs/index.rst
b/providers/common/ai/docs/index.rst
index c01522e37f7..014d49450f1 100644
--- a/providers/common/ai/docs/index.rst
+++ b/providers/common/ai/docs/index.rst
@@ -19,15 +19,39 @@
``apache-airflow-providers-common-ai``
##################################################
+The ``common.ai`` provider is the vendor-neutral way to put LLM and agent
steps in a Dag.
+
When to use this provider
--------------------------
-``common.ai`` is the vendor-neutral way to put LLM and agent steps in a Dag.
It is built on
-`pydantic-ai <https://ai.pydantic.dev/>`__, so the model vendor (OpenAI,
Anthropic, Google,
-Bedrock, …) is picked by the connection ``llm_conn_id`` points at — switching
providers later
-is a connection change, not a Dag rewrite. The AI step is orchestrated by
Airflow: the model
-calls, the agent loop, and any tools all run in the Airflow worker, where they
get retries,
-logging, and observability like any other task.
+.. list-table::
+ :header-rows: 1
+ :widths: 40 30 30
+
+ * - Use case
+ - Use
+ - Package
+ * - Portable generation, classification, extraction, branching, or a
+ worker-run agent with toolsets
+ - ``common.ai``
+ - ``apache-airflow-providers-common-ai``
+ * - A vendor's native Embeddings, Responses, or Batch API
+ - The vendor's own provider
+ - e.g. :doc:`apache-airflow-providers-openai:index`,
+ :doc:`apache-airflow-providers-anthropic:index`,
+ :doc:`apache-airflow-providers-cohere:index`
+ * - A vendor-managed, server-side agent session (e.g. Anthropic Managed
Agents)
+ - The vendor's own provider
+ - e.g. :doc:`apache-airflow-providers-anthropic:index`
+
+``common.ai`` is built on `pydantic-ai <https://ai.pydantic.dev/>`__, so the
model vendor
+(OpenAI, Anthropic, Google, Bedrock, …) is picked by the connection
``llm_conn_id`` points
+at — switching providers later is a connection change, not a Dag rewrite.
Existing LangChain
+tools aren't locked out either: with the ``langchain`` extra installed, they
can be wrapped
+in ``LangChainToolset`` and dropped straight into a common.ai agent (see
:doc:`toolsets`).
+The AI step is orchestrated by Airflow: the model calls, the agent loop, and
any tools all
+run in the Airflow worker, where they get retries, logging, and observability
like any other
+task.
Use it when a Dag needs:
@@ -35,8 +59,9 @@ Use it when a Dag needs:
:doc:`LLMOperator and @task.llm <operators/llm>`, with Pydantic-typed output
pushed to XCom.
* **Branching on a model's decision** — :doc:`LLMBranchOperator
<operators/llm_branch>`.
* **Agents with tools** — :doc:`AgentOperator <operators/agent>` runs a
multi-turn agent loop
- in the worker, calling Airflow-defined toolsets (SQL, hooks, MCP servers),
with optional
- human-in-the-loop review and durable step replay.
+ in the worker, calling Airflow-defined :doc:`toolsets <toolsets>` (SQL,
hooks, MCP servers),
+ with optional human-in-the-loop review and durable step replay — if the task
retries after
+ a failure, completed steps are replayed from cache instead of re-executing.
* **Document pipelines** — loading, file analysis, embeddings, and retrieval
for RAG
(see :doc:`operators/index`).
@@ -47,6 +72,7 @@ a service the vendor runs for you, which no vendor-neutral
operator wraps:
* :doc:`apache-airflow-providers-anthropic:index` — the Claude Message Batches
API, and
Managed Agents sessions where the agent loop runs on Anthropic's
infrastructure rather
than in the Airflow worker.
+* :doc:`apache-airflow-providers-cohere:index` — Cohere's own Embed API.
As a rule of thumb: if Airflow should *run* the AI step (and the model should
stay
swappable), use ``common.ai``; if the Dag *submits work to* a vendor-managed
service and
diff --git a/providers/openai/docs/index.rst b/providers/openai/docs/index.rst
index 21dd8a9b3dd..678075f53e3 100644
--- a/providers/openai/docs/index.rst
+++ b/providers/openai/docs/index.rst
@@ -19,20 +19,25 @@
``apache-airflow-providers-openai``
======================================
+The ``openai`` provider gives Dags direct access to OpenAI's own APIs — this
page compares
+that choice against ``common.ai``.
+
When to use this provider
--------------------------
Use ``openai`` when a Dag needs OpenAI's native API surface — thin wrappers
over
-OpenAI-specific endpoints and options:
+OpenAI-specific endpoints and options, built on ``OpenAIHook``, the underlying
client the
+operators below share:
* ``OpenAIEmbeddingOperator`` — call the Embeddings API directly, e.g. to feed
a vector
store.
* ``OpenAIResponseOperator`` — call the
`Responses API <https://platform.openai.com/docs/api-reference/responses>`__
with
OpenAI-specific parameters.
-* ``OpenAITriggerBatchOperator`` and ``OpenAIHook`` — submit a
+* ``OpenAITriggerBatchOperator`` — submit a
`Batch API <https://platform.openai.com/docs/guides/batch>`__ job for
asynchronous bulk
- processing and wait for it to complete.
+ processing and wait for it to complete; OpenAI prices Batch API calls at
roughly half the
+ cost of the equivalent synchronous call, in exchange for a turnaround of up
to ~24 hours.
Use :doc:`apache-airflow-providers-common-ai:index` instead when the AI step
should be run by
Airflow itself and stay vendor-neutral:
