kaxil commented on code in PR #77073:
URL: https://github.com/apache/airflow/pull/77073#discussion_r4622778132


##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -824,8 +824,18 @@ def llm_hook(self) -> PydanticAIHook:
         }
         return PydanticAIHook.get_hook(self.llm_conn_id, 
hook_params=hook_params)
 
-    def _build_agent(self) -> Agent[object, Any]:
-        """Build and return a pydantic-ai Agent from the operator's config."""
+    def build_agent(self) -> Agent[object, Any]:

Review Comment:
   Main has since changed this to `_build_agent(self, *, 
approval_context=None)` (#76882): `execute()` passes the task context so a 
durable run can pause for tool approval. When you rebase, can `build_agent()` 
stay argument-free and call the private `_build_agent()`, with `execute()` 
still calling the private one? If `execute()` calls 
`build_agent(approval_context=context)` instead, that kwarg becomes public and 
a subclass overriding `build_agent(self)` gets a `TypeError` on every run. The 
docstring should then also say that for a durable task the returned agent has 
no tool-approval step.



##########
providers/common/ai/src/airflow/providers/common/ai/operators/llm.py:
##########
@@ -389,14 +389,28 @@ def _create_agent(self, *, output_type: Any, 
instructions: str) -> Agent[object,
             retries=self.output_retries,
         )
 
+    def build_agent(self) -> Agent[object, Any]:

Review Comment:
   A third-party `LLMOperator` subclass that builds its agent inside 
`execute()`, the way the four in-tree ones did, now inherits a `build_agent()` 
that returns a different agent from the one the task runs, with no error. Could 
the docstring say a subclass that builds its agent differently should override 
this and call it from `execute()`?



##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
 
 The test above scripts the model's replies, so it can't tell you whether a 
changed
 prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model 
and
-a library such as `pydantic-evals 
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you 
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the 
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+    # evals/eval_orders_report.py
+    from pydantic_evals import Case, Dataset
+    from pydantic_evals.evaluators import Contains
+
+    from orders_report import orders_report  # dags/orders_report.py from above

Review Comment:
   Run as a plain script, this import only resolves with `dags/` on the path, 
and the `pythonpath` note above only covers pytest. Could you show the command, 
e.g. `PYTHONPATH=dags python evals/eval_orders_report.py`?



##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
 
 The test above scripts the model's replies, so it can't tell you whether a 
changed
 prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model 
and
-a library such as `pydantic-evals 
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you 
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its

Review Comment:
   This doesn't hold for a mapped task: `dag.get_task()` returns a 
`MappedOperator` there, which has no `build_agent()`, so it's an 
`AttributeError`. Mapped `@task.agent` / `@task.llm` are a documented pattern, 
so can this say they aren't covered?



##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
 
 The test above scripts the model's replies, so it can't tell you whether a 
changed
 prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model 
and
-a library such as `pydantic-evals 
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you 
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the 
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+    # evals/eval_orders_report.py
+    from pydantic_evals import Case, Dataset
+    from pydantic_evals.evaluators import Contains
+
+    from orders_report import orders_report  # dags/orders_report.py from above
+
+    agent = orders_report().get_task("orders_question").build_agent()
+
+    dataset = Dataset(
+        name="orders_question",
+        cases=[Case(inputs="How many orders are there?", 
evaluators=[Contains("3")])],
+    )
+    report = dataset.evaluate_sync(lambda question: 
agent.run_sync(question).output)
+    report.print()
+
+``build_agent()`` runs outside a task, so templates aren't rendered and a 
templated

Review Comment:
   This applies to every template field, not only `system_prompt`. A templated 
`llm_conn_id`, `model_id` or `model_settings` also reaches the agent as 
written, and a templated conn id fails at connection lookup. Could this (and 
the two docstrings) say template fields are used as written?



##########
providers/common/ai/tests/unit/common/ai/operators/test_llm.py:
##########
@@ -577,6 +580,126 @@ def _make_context(ti_id=None):
     return MagicMock(**{"__getitem__": lambda self, key: {"task_instance": 
ti}[key]})
 
 
+class TestLLMOperatorBuildAgent:
+    """``build_agent()`` returns the agent ``execute`` runs, so an eval can 
run a task's own agent."""
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def test_execute_runs_the_agent_build_agent_returns(self, mock_hook_cls, 
make_mock_run_result):
+        create_agent = mock_hook_cls.get_hook.return_value.create_agent
+        mock_agent = MagicMock(spec=["run_sync", "instrument"])
+        mock_agent.run_sync.return_value = make_mock_run_result("ok")
+        create_agent.return_value = mock_agent
+        op = LLMOperator(
+            task_id="t",
+            prompt="p",
+            llm_conn_id="my_llm",
+            system_prompt="Be brief.",
+            model_settings={"temperature": 0},
+            output_retries=2,
+        )
+        expected = call(
+            output_type=str, instructions="Be brief.", 
model_settings={"temperature": 0}, retries=2
+        )
+
+        assert op.build_agent() is mock_agent
+        op.execute(context=_make_context(ti_id="ti-1"))
+
+        assert create_agent.call_args_list == [expected, expected]
+        mock_agent.run_sync.assert_called_once()
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def 
test_build_agent_on_a_task_from_a_dag_passes_a_templated_system_prompt_as_written(
+        self, mock_hook_cls
+    ):
+        with DAG("d", schedule=None) as dag:
+            LLMOperator(
+                task_id="summarize", prompt="p", llm_conn_id="my_llm", 
system_prompt="Summarize {{ ds }}."
+            )
+
+        dag.get_task("summarize").build_agent()
+
+        create_agent = mock_hook_cls.get_hook.return_value.create_agent
+        assert create_agent.call_args.kwargs["instructions"] == "Summarize {{ 
ds }}."
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def test_build_agent_on_a_decorated_task(self, mock_hook_cls):
+        with DAG("d", schedule=None) as dag:
+
+            @task.llm(llm_conn_id="my_llm", system_prompt="Be brief.", 
output_type=Summary)
+            def summarize() -> str:
+                return "p"
+
+            summarize()
+
+        dag.get_task("summarize").build_agent()
+
+        kwargs = 
mock_hook_cls.get_hook.return_value.create_agent.call_args.kwargs
+        assert kwargs["output_type"] is Summary
+        assert kwargs["instructions"] == "Be brief."
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def 
test_branch_operator_builds_an_agent_that_picks_a_downstream_task(self, 
mock_hook_cls):
+        op = LLMBranchOperator(task_id="route", prompt="p", 
llm_conn_id="my_llm", system_prompt="Route it.")
+        op.downstream_task_ids = {"billing", "support"}
+
+        op.build_agent()
+
+        kwargs = 
mock_hook_cls.get_hook.return_value.create_agent.call_args.kwargs
+        assert set(TypeAdapter(kwargs["output_type"]).json_schema()["enum"]) 
== {"billing", "support"}
+        assert kwargs["instructions"] == "Route it."
+
+    def 
test_branch_operator_without_downstream_tasks_cannot_build_an_agent(self):
+        op = LLMBranchOperator(task_id="route", prompt="p", 
llm_conn_id="my_llm")
+
+        with pytest.raises(ValueError, match="has no downstream tasks"):
+            op.build_agent()
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def 
test_file_analysis_operator_builds_an_agent_with_its_system_prompt(self, 
mock_hook_cls):
+        op = LLMFileAnalysisOperator(
+            task_id="t", prompt="p", llm_conn_id="my_llm", 
file_path="/tmp/app.log", output_type=Summary
+        )
+
+        op.build_agent()
+
+        kwargs = 
mock_hook_cls.get_hook.return_value.create_agent.call_args.kwargs
+        assert kwargs["output_type"] is Summary
+        assert kwargs["instructions"] == op._build_system_prompt()
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def 
test_sql_operator_builds_an_agent_with_the_schema_in_its_system_prompt(self, 
mock_hook_cls):
+        op = LLMSQLQueryOperator(task_id="t", prompt="p", llm_conn_id="my_llm")
+        schema = "Table: orders\nColumns: id INTEGER"
+
+        with patch.object(LLMSQLQueryOperator, "_get_schema_context", 
return_value=schema):

Review Comment:
   Can this `patch.object` (and the one on line 695) take `autospec=True` like 
the other patches in this class?



##########
providers/common/ai/tests/unit/common/ai/operators/test_llm.py:
##########
@@ -577,6 +580,126 @@ def _make_context(ti_id=None):
     return MagicMock(**{"__getitem__": lambda self, key: {"task_instance": 
ti}[key]})
 
 
+class TestLLMOperatorBuildAgent:
+    """``build_agent()`` returns the agent ``execute`` runs, so an eval can 
run a task's own agent."""
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def test_execute_runs_the_agent_build_agent_returns(self, mock_hook_cls, 
make_mock_run_result):
+        create_agent = mock_hook_cls.get_hook.return_value.create_agent
+        mock_agent = MagicMock(spec=["run_sync", "instrument"])
+        mock_agent.run_sync.return_value = make_mock_run_result("ok")
+        create_agent.return_value = mock_agent
+        op = LLMOperator(
+            task_id="t",
+            prompt="p",
+            llm_conn_id="my_llm",
+            system_prompt="Be brief.",
+            model_settings={"temperature": 0},
+            output_retries=2,
+        )
+        expected = call(
+            output_type=str, instructions="Be brief.", 
model_settings={"temperature": 0}, retries=2
+        )
+
+        assert op.build_agent() is mock_agent
+        op.execute(context=_make_context(ti_id="ti-1"))
+
+        assert create_agent.call_args_list == [expected, expected]
+        mock_agent.run_sync.assert_called_once()
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def 
test_build_agent_on_a_task_from_a_dag_passes_a_templated_system_prompt_as_written(
+        self, mock_hook_cls
+    ):
+        with DAG("d", schedule=None) as dag:
+            LLMOperator(
+                task_id="summarize", prompt="p", llm_conn_id="my_llm", 
system_prompt="Summarize {{ ds }}."
+            )
+
+        dag.get_task("summarize").build_agent()
+
+        create_agent = mock_hook_cls.get_hook.return_value.create_agent
+        assert create_agent.call_args.kwargs["instructions"] == "Summarize {{ 
ds }}."
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def test_build_agent_on_a_decorated_task(self, mock_hook_cls):
+        with DAG("d", schedule=None) as dag:
+
+            @task.llm(llm_conn_id="my_llm", system_prompt="Be brief.", 
output_type=Summary)
+            def summarize() -> str:
+                return "p"
+
+            summarize()
+
+        dag.get_task("summarize").build_agent()
+
+        kwargs = 
mock_hook_cls.get_hook.return_value.create_agent.call_args.kwargs
+        assert kwargs["output_type"] is Summary
+        assert kwargs["instructions"] == "Be brief."
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def 
test_branch_operator_builds_an_agent_that_picks_a_downstream_task(self, 
mock_hook_cls):
+        op = LLMBranchOperator(task_id="route", prompt="p", 
llm_conn_id="my_llm", system_prompt="Route it.")
+        op.downstream_task_ids = {"billing", "support"}
+
+        op.build_agent()
+
+        kwargs = 
mock_hook_cls.get_hook.return_value.create_agent.call_args.kwargs
+        assert set(TypeAdapter(kwargs["output_type"]).json_schema()["enum"]) 
== {"billing", "support"}
+        assert kwargs["instructions"] == "Route it."
+
+    def 
test_branch_operator_without_downstream_tasks_cannot_build_an_agent(self):
+        op = LLMBranchOperator(task_id="route", prompt="p", 
llm_conn_id="my_llm")
+
+        with pytest.raises(ValueError, match="has no downstream tasks"):
+            op.build_agent()
+
+    @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", 
autospec=True)
+    def 
test_file_analysis_operator_builds_an_agent_with_its_system_prompt(self, 
mock_hook_cls):

Review Comment:
   Unlike the `LLMOperator` test above, nothing checks that the file-analysis 
`execute()` runs the agent `build_agent()` returns. The existing file-analysis 
tests only assert the `create_agent` kwarg names, so `execute()` building its 
own agent with different instructions stays green. Could this also call 
`execute()` and compare both `create_agent` calls?



##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
 
 The test above scripts the model's replies, so it can't tell you whether a 
changed
 prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model 
and
-a library such as `pydantic-evals 
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you 
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the 
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+    # evals/eval_orders_report.py
+    from pydantic_evals import Case, Dataset
+    from pydantic_evals.evaluators import Contains
+
+    from orders_report import orders_report  # dags/orders_report.py from above
+
+    agent = orders_report().get_task("orders_question").build_agent()
+
+    dataset = Dataset(
+        name="orders_question",
+        cases=[Case(inputs="How many orders are there?", 
evaluators=[Contains("3")])],
+    )
+    report = dataset.evaluate_sync(lambda question: 
agent.run_sync(question).output)
+    report.print()
+
+``build_agent()`` runs outside a task, so templates aren't rendered and a 
templated
+``system_prompt`` reaches the agent as written. ``@task.llm``, ``LLMOperator`` 
and
+the other LLM operators have it too; for ``LLMSQLQueryOperator`` and

Review Comment:
   `LLMBatchOperator` is one of the LLM operators but has no `build_agent()`, 
since it subclasses `BaseOperator`. Can this name the operators that do?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to