aarmens702-hub commented on code in PR #77073:
URL: https://github.com/apache/airflow/pull/77073#discussion_r4638755188


##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -824,8 +824,18 @@ def llm_hook(self) -> PydanticAIHook:
         }
         return PydanticAIHook.get_hook(self.llm_conn_id, 
hook_params=hook_params)
 
-    def _build_agent(self) -> Agent[object, Any]:
-        """Build and return a pydantic-ai Agent from the operator's config."""
+    def build_agent(self) -> Agent[object, Any]:

Review Comment:
   Done. Main's `_build_agent(self, *, approval_context=None)` is unchanged. 
`execute()` still calls it with the task context. `build_agent()` takes no 
arguments and returns `self._build_agent()`, so a subclass that overrides 
`_build_agent` gets the same agent from both. The docstring says a durable 
task's agent from `build_agent()` has no tool-approval step, and a new test 
checks that `build_agent()` passes no approval context.
   
   Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before 
posting
   



##########
providers/common/ai/src/airflow/providers/common/ai/operators/llm.py:
##########
@@ -389,14 +389,28 @@ def _create_agent(self, *, output_type: Any, 
instructions: str) -> Agent[object,
             retries=self.output_retries,
         )
 
+    def build_agent(self) -> Agent[object, Any]:

Review Comment:
   Added: "A subclass that builds its agent differently should override this 
and call it from `execute`, so both return the same agent."
   
   Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before 
posting
   



##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
 
 The test above scripts the model's replies, so it can't tell you whether a 
changed
 prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model 
and
-a library such as `pydantic-evals 
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you 
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its

Review Comment:
   Added a sentence saying mapped tasks aren't covered: `get_task()` returns a 
`MappedOperator` for them, which has no `build_agent()`.
   
   Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before 
posting
   



##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
 
 The test above scripts the model's replies, so it can't tell you whether a 
changed
 prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model 
and
-a library such as `pydantic-evals 
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you 
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the 
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+    # evals/eval_orders_report.py
+    from pydantic_evals import Case, Dataset
+    from pydantic_evals.evaluators import Contains
+
+    from orders_report import orders_report  # dags/orders_report.py from above

Review Comment:
   Added the command: `PYTHONPATH=dags python evals/eval_orders_report.py`, run 
from the project root.
   
   Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before 
posting
   



##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
 
 The test above scripts the model's replies, so it can't tell you whether a 
changed
 prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model 
and
-a library such as `pydantic-evals 
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you 
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the 
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+    # evals/eval_orders_report.py
+    from pydantic_evals import Case, Dataset
+    from pydantic_evals.evaluators import Contains
+
+    from orders_report import orders_report  # dags/orders_report.py from above
+
+    agent = orders_report().get_task("orders_question").build_agent()
+
+    dataset = Dataset(
+        name="orders_question",
+        cases=[Case(inputs="How many orders are there?", 
evaluators=[Contains("3")])],
+    )
+    report = dataset.evaluate_sync(lambda question: 
agent.run_sync(question).output)
+    report.print()
+
+``build_agent()`` runs outside a task, so templates aren't rendered and a 
templated

Review Comment:
   The docs and both docstrings now say every template field is used as written 
outside a task run, naming `system_prompt`, `llm_conn_id` and `model_id`, and 
that a templated `llm_conn_id` fails at connection lookup.
   
   Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before 
posting
   



##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
 
 The test above scripts the model's replies, so it can't tell you whether a 
changed
 prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model 
and
-a library such as `pydantic-evals 
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you 
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the 
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+    # evals/eval_orders_report.py
+    from pydantic_evals import Case, Dataset
+    from pydantic_evals.evaluators import Contains
+
+    from orders_report import orders_report  # dags/orders_report.py from above
+
+    agent = orders_report().get_task("orders_question").build_agent()
+
+    dataset = Dataset(
+        name="orders_question",
+        cases=[Case(inputs="How many orders are there?", 
evaluators=[Contains("3")])],
+    )
+    report = dataset.evaluate_sync(lambda question: 
agent.run_sync(question).output)
+    report.print()
+
+``build_agent()`` runs outside a task, so templates aren't rendered and a 
templated
+``system_prompt`` reaches the agent as written. ``@task.llm``, ``LLMOperator`` 
and
+the other LLM operators have it too; for ``LLMSQLQueryOperator`` and

Review Comment:
   The docs now list the six operators that have it, each with its decorator: 
`AgentOperator`, `LLMOperator`, `LLMBranchOperator`, `LLMFileAnalysisOperator`, 
`LLMSQLQueryOperator` and `LLMSchemaCompareOperator`. `LLMBatchOperator` sends 
requests through the provider batch APIs and never builds a pydantic-ai agent, 
so it has nothing to return.
   
   Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before 
posting
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to