This is an automated email from the ASF dual-hosted git repository.
kaxil pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new ab2677888ce Lead the common.ai docs with a runnable quick start and a
verifiable result (#73556)
ab2677888ce is described below
commit ab2677888ce51b073798af18a7a5f9b14ea65a3d
Author: Kaxil Naik <[email protected]>
AuthorDate: Tue Sep 22 15:56:00 2026 +0100
Lead the common.ai docs with a runnable quick start and a verifiable result
(#73556)
The landing page opened like a package reference, and the quick start
stopped at "run it like any other Dag" without saying where the answer
lands or what a successful run looks like. The first code a reader saw
asked a model to summarize a report the snippet never supplied.
The landing page now opens with what the provider does in Airflow terms,
three starting links, and the same two-task Dag the quick start runs. The
quick start gains the missing steps: where to save the file, how to trigger
the Dag, and what to look for in the logs and the XCom tab. The example Dag
supplies its input in the file and adds a downstream task that consumes the
model's answer, so the Airflow integration is visible in the first example.
---
providers/common/ai/docs/index.rst | 48 +++++----
providers/common/ai/docs/quickstart.rst | 120 +++++++++++++--------
.../common/ai/example_dags/example_quickstart.py | 25 +++--
3 files changed, 119 insertions(+), 74 deletions(-)
diff --git a/providers/common/ai/docs/index.rst
b/providers/common/ai/docs/index.rst
index d2c0a4fee2f..8b1bbba198d 100644
--- a/providers/common/ai/docs/index.rst
+++ b/providers/common/ai/docs/index.rst
@@ -19,10 +19,34 @@
``apache-airflow-providers-common-ai``
##################################################
-The ``common.ai`` provider is the vendor-neutral way to put LLM and agent
steps in a Dag.
+Run model calls and tool-using agents as Airflow tasks. A task can classify,
extract,
+summarize or route with any model vendor, or hand a model a set of tools built
from your
+Airflow connections and let it work. Airflow supplies what a script does not:
the API key
+comes from a connection, a failed call retries, a run can pause for a person
to approve the
+output, the result lands in XCom for the next task, and the whole thing runs
on a schedule.
+
+Start here
+----------
+
+- :doc:`quickstart` — install, connect a vendor, run a two-task Dag and check
its output.
+- :doc:`use_cases/index` — ten jobs a data team already has, each with the Dag
that does it.
+- :doc:`model_providers` — which vendors work, and the extra, connection and
prefix for each.
+
+This is the Dag the quick start runs. The ``summarize`` task sends the release
notes to the
+model on the ``pydanticai_default`` connection; ``publish`` receives the
answer like any
+other upstream result:
+
+.. exampleinclude::
/../../ai/src/airflow/providers/common/ai/example_dags/example_quickstart.py
+ :language: python
+ :start-after: [START howto_quickstart_llm]
+ :end-before: [END howto_quickstart_llm]
+
+Point ``pydanticai_default`` at OpenAI, Anthropic, Google, Bedrock or a
self-hosted server
+and the Dag does not change. :doc:`concepts` explains the ideas behind the
provider in one
+page.
When to use this provider
---------------------------
+-------------------------
.. list-table::
:header-rows: 1
@@ -49,30 +73,10 @@ When to use this provider
- The vendor's own provider
- e.g. :doc:`apache-airflow-providers-anthropic:index`
-``common.ai`` is built on `pydantic-ai <https://ai.pydantic.dev/>`__: the
connection picks the
-model vendor, and Airflow runs the AI step like any other task.
:doc:`concepts` explains the
-ideas behind the provider in one page; :doc:`operators/index` lists what each
operator is for.
-
As a rule of thumb: if Airflow should *run* the AI step (and the model should
stay
swappable), use ``common.ai``; if the Dag *submits work to* a vendor-managed
service and
waits for the result, use that vendor's provider.
-For example, this ``LLMOperator`` call is unchanged whether ``llm_conn_id``
points at an
-OpenAI, Anthropic, or other pydantic-ai-supported connection:
-
-.. exampleinclude::
/../../ai/src/airflow/providers/common/ai/example_dags/example_llm.py
- :language: python
- :start-after: [START howto_operator_llm_basic]
- :end-before: [END howto_operator_llm_basic]
-
-Getting started
----------------
-
-* :doc:`installation` — which extra to install for your model vendor.
-* :doc:`quickstart` — a connection and a first ``@task.llm`` in three steps.
-* :doc:`concepts` — connections, operators, toolsets, hooks and XCom in one
page.
-* :doc:`use_cases/index` — jobs a data team already has, each with the Dag
that does it.
-
.. toctree::
:titlesonly:
:hidden:
diff --git a/providers/common/ai/docs/quickstart.rst
b/providers/common/ai/docs/quickstart.rst
index 38f0685bad9..ba9ca66db34 100644
--- a/providers/common/ai/docs/quickstart.rst
+++ b/providers/common/ai/docs/quickstart.rst
@@ -20,74 +20,104 @@
Quick start
===========
-This guide installs the provider, configures a connection, and runs a first
-LLM task.
+In five steps you install the provider, connect it to a model vendor, and run
a Dag in
+which one task asks a model to summarize release notes and a second task uses
the answer.
+At the end you know where the model's output lands and what a successful run
looks like.
-Before you start: this assumes a working
:doc:`apache-airflow:installation/index`
-(Airflow 3.0+) already exists, you have an API key for the LLM provider you
-plan to use, and step 3 below makes a real, billed API call to that provider.
+You need a working :doc:`Airflow installation
<apache-airflow:installation/index>` on
+Airflow 3.0 or later and an API key for the model vendor you plan to use. Step
4 makes one
+real, billed API call.
-1. Install
-----------
+1. Install the provider
+-----------------------
-Install the provider together with the extra matching the model SDK you plan
-to use — ``openai``, ``anthropic``, ``google``, or ``bedrock`` (see
-:doc:`installation` for the full list of available extras). Replace ``<extra>``
-below with the one you need:
+Install the extra that matches your model vendor. The example below uses
OpenAI; swap the
+extra for ``anthropic``, ``google`` or ``bedrock`` if that is what you have
+(:doc:`model_providers` lists every vendor):
.. code-block:: bash
- pip install "apache-airflow-providers-common-ai[<extra>]"
+ pip install "apache-airflow-providers-common-ai[openai]"
-2. Configure the connection
-----------------------------
+2. Create the connection
+------------------------
-Every LLM call goes through a Pydantic AI connection (``conn_type``
``pydanticai``,
-default connection id ``pydanticai_default``). The model is set in
``provider:model``
-format and the API key goes in the password field. See
:ref:`howto/connection:pydanticai`
-for the full reference, including providers that
-don't need an API key (Bedrock, Vertex AI).
+Model calls go through an Airflow connection of type ``pydanticai``. The
connection holds
+the API key and the model name in ``provider:model`` form, so switching
vendors later is a
+connection change, not a Dag change. The Dag below uses the default connection
id,
+``pydanticai_default``.
-The quickest way to set one up is an environment variable. Replace
-``openai:gpt-5.6-sol`` with a model you have access to and ``sk-...`` with your
-actual API key:
+The quickest way to create it is an environment variable on the machine that
runs the
+scheduler and the workers. Replace ``sk-...`` with your key and
``openai:gpt-5.6-sol`` with
+a model you have access to:
.. code-block:: bash
export AIRFLOW_CONN_PYDANTICAI_DEFAULT='{"conn_type": "pydanticai",
"password": "sk-...", "extra": {"model": "openai:gpt-5.6-sol"}}'
-Or add it through the Airflow UI (``Admin > Connections``) or the CLI
(``airflow connections add``).
+You can also create it in the UI under **Admin > Connections**: choose the
connection type
+**Pydantic AI**, set the connection id to ``pydanticai_default``, put the API
key in
+**Password** and the model in the **Model** field. Vendors that authenticate
through the
+environment instead of a key (Bedrock, Vertex AI) have their own connection
types; see
+:doc:`connections/pydantic_ai`.
-3. Write your first Dag
-------------------------
+3. Save the Dag
+---------------
-The ``@task.llm`` decorator turns a function that returns a prompt string into
-a task that sends that prompt to the LLM and returns its response:
+Save the following as ``quickstart_llm.py`` in your Dags folder, the directory
+``[core] dags_folder`` points at (``$AIRFLOW_HOME/dags`` by default):
.. exampleinclude::
/../../ai/src/airflow/providers/common/ai/example_dags/example_quickstart.py
:language: python
:start-after: [START howto_quickstart_llm]
:end-before: [END howto_quickstart_llm]
-Run it like any other Dag (``airflow dags test quickstart_llm``) and the
-``summarize`` task pushes the LLM's response to XCom.
+Two tasks. ``summarize`` is a ``@task.llm`` task: the function returns the
prompt, the
+decorator sends it to the model on ``pydanticai_default`` and pushes the
model's reply to
+XCom as the task's return value. ``publish`` is an ordinary ``@task`` that
receives that
+reply as its argument, the same way any Airflow task receives an upstream
result, logs it,
+and returns a small dict of its own.
+
+4. Run it
+---------
+
+Run the Dag once in the foreground from the machine where you saved the file:
+
+.. code-block:: bash
+
+ airflow dags test quickstart_llm
+
+The command parses the Dag, runs ``summarize`` and then ``publish``, and
prints both task
+logs to the terminal. You can also trigger the Dag from the UI with the play
button on the
+``quickstart_llm`` row of the Dags list.
+
+5. Check the result
+-------------------
+
+In the terminal output, or in each task's log in the UI, look for two lines:
+
+- From ``summarize``, a line starting ``LLM run complete`` with the model name
and the
+ token counts. This is the provider's post-run summary, and it appears after
every model
+ call.
+- From ``publish``, a line starting ``Release summary:`` followed by the two
sentences the
+ model wrote.
-Structured output
-^^^^^^^^^^^^^^^^^^
+In the UI, open the run in the Grid view, select the ``summarize`` task and
open its
+**XCom** tab: the ``return_value`` entry holds the model's reply. The
``publish`` task's
+``return_value`` is the dict with the summary and its length.
-Need typed data instead of a string? Set ``output_type`` to a Pydantic
-``BaseModel`` and the model instance is pushed to XCom unchanged. See
-:doc:`structured_output` for the full example and its XCom-deserialization
-requirements.
+If the run fails before the model is called, the error names what is missing: a
+connection without a model, a model name without a ``provider:`` prefix, or a
vendor
+package that is not installed. :doc:`troubleshooting` lists each message with
its fix.
Where to go next
------------------
-
-- :doc:`use_cases/index` — start here for what to build.
-- :doc:`operators/index` — the full set of operators and ``@task`` decorators
- (file analysis, SQL, branching, schema comparison).
-- :doc:`toolsets/index` — give an agent tools built from Airflow hooks, SQL
- databases, or MCP servers.
-- :ref:`howto/operator:agent` — run a multi-turn agent that reasons and calls
- tools instead of a single prompt-response call.
-- :doc:`observability` — trace LLM and tool calls with OpenTelemetry.
+----------------
+
+- :doc:`use_cases/index` shows jobs a data team already has, each with the Dag
that does
+ it.
+- :doc:`structured_output` returns a typed Pydantic object instead of a
string, so the
+ downstream task gets fields rather than prose.
+- :doc:`operators/index` picks the operator for a job: branching on an answer,
analyzing
+ files, generating SQL, batch processing.
+- :doc:`operators/agent` gives the model tools built from Airflow hooks, SQL
databases or
+ MCP servers, so it can act instead of only answering.
diff --git
a/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_quickstart.py
b/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_quickstart.py
index 478cb332e68..45fea2f59fb 100644
---
a/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_quickstart.py
+++
b/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_quickstart.py
@@ -14,23 +14,34 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-"""Quickstart example: a first @task.llm Dag."""
+"""Quickstart example: a first ``@task.llm`` Dag with a downstream task that
uses the answer."""
from __future__ import annotations
# [START howto_quickstart_llm]
from airflow.sdk import dag, task
+RELEASE_NOTES = """
+Changes since the last release:
+- Tasks can now carry a retry policy that decides whether a failure is worth
retrying.
+- Heartbeat writes are batched, which cut metadata database load by about a
third in testing.
+- Fixed a crash when two tasks in one Dag file shared a task id.
+- Dropped support for Python 3.9.
+"""
+
@dag(schedule=None, tags=["example"])
def quickstart_llm():
- @task.llm(llm_conn_id="pydanticai_default", system_prompt="You are a
helpful assistant. Be concise.")
- def summarize(text: str):
- return f"Summarize this article: {text}"
+ @task.llm(llm_conn_id="pydanticai_default", system_prompt="You write
release announcements. Be concise.")
+ def summarize(notes: str):
+ return f"Summarize these release notes in two sentences for a team
status update:\n{notes}"
+
+ @task
+ def publish(summary: str) -> dict[str, str | int]:
+ print(f"Release summary: {summary}")
+ return {"summary": summary, "characters": len(summary)}
- summarize(
- "Apache Airflow is a platform for programmatically authoring,
scheduling, and monitoring workflows."
- )
+ publish(summarize(RELEASE_NOTES))
quickstart_llm()
