kaxil commented on code in PR #77073:
URL: https://github.com/apache/airflow/pull/77073#discussion_r4622778132
##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -824,8 +824,18 @@ def llm_hook(self) -> PydanticAIHook:
}
return PydanticAIHook.get_hook(self.llm_conn_id,
hook_params=hook_params)
- def _build_agent(self) -> Agent[object, Any]:
- """Build and return a pydantic-ai Agent from the operator's config."""
+ def build_agent(self) -> Agent[object, Any]:
Review Comment:
Main has since changed this to `_build_agent(self, *,
approval_context=None)` (#76882): `execute()` passes the task context so a
durable run can pause for tool approval. When you rebase, can `build_agent()`
stay argument-free and call the private `_build_agent()`, with `execute()`
still calling the private one? If `execute()` calls
`build_agent(approval_context=context)` instead, that kwarg becomes public and
a subclass overriding `build_agent(self)` gets a `TypeError` on every run. The
docstring should then also say that for a durable task the returned agent has
no tool-approval step.
##########
providers/common/ai/src/airflow/providers/common/ai/operators/llm.py:
##########
@@ -389,14 +389,28 @@ def _create_agent(self, *, output_type: Any,
instructions: str) -> Agent[object,
retries=self.output_retries,
)
+ def build_agent(self) -> Agent[object, Any]:
Review Comment:
A third-party `LLMOperator` subclass that builds its agent inside
`execute()`, the way the four in-tree ones did, now inherits a `build_agent()`
that returns a different agent from the one the task runs, with no error. Could
the docstring say a subclass that builds its agent differently should override
this and call it from `execute()`?
##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
The test above scripts the model's replies, so it can't tell you whether a
changed
prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model
and
-a library such as `pydantic-evals
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+ # evals/eval_orders_report.py
+ from pydantic_evals import Case, Dataset
+ from pydantic_evals.evaluators import Contains
+
+ from orders_report import orders_report # dags/orders_report.py from above
Review Comment:
Run as a plain script, this import only resolves with `dags/` on the path,
and the `pythonpath` note above only covers pytest. Could you show the command,
e.g. `PYTHONPATH=dags python evals/eval_orders_report.py`?
##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
The test above scripts the model's replies, so it can't tell you whether a
changed
prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model
and
-a library such as `pydantic-evals
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
Review Comment:
This doesn't hold for a mapped task: `dag.get_task()` returns a
`MappedOperator` there, which has no `build_agent()`, so it's an
`AttributeError`. Mapped `@task.agent` / `@task.llm` are a documented pattern,
so can this say they aren't covered?
##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
The test above scripts the model's replies, so it can't tell you whether a
changed
prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model
and
-a library such as `pydantic-evals
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+ # evals/eval_orders_report.py
+ from pydantic_evals import Case, Dataset
+ from pydantic_evals.evaluators import Contains
+
+ from orders_report import orders_report # dags/orders_report.py from above
+
+ agent = orders_report().get_task("orders_question").build_agent()
+
+ dataset = Dataset(
+ name="orders_question",
+ cases=[Case(inputs="How many orders are there?",
evaluators=[Contains("3")])],
+ )
+ report = dataset.evaluate_sync(lambda question:
agent.run_sync(question).output)
+ report.print()
+
+``build_agent()`` runs outside a task, so templates aren't rendered and a
templated
Review Comment:
This applies to every template field, not only `system_prompt`. A templated
`llm_conn_id`, `model_id` or `model_settings` also reaches the agent as
written, and a templated conn id fails at connection lookup. Could this (and
the two docstrings) say template fields are used as written?
##########
providers/common/ai/tests/unit/common/ai/operators/test_llm.py:
##########
@@ -577,6 +580,126 @@ def _make_context(ti_id=None):
return MagicMock(**{"__getitem__": lambda self, key: {"task_instance":
ti}[key]})
+class TestLLMOperatorBuildAgent:
+ """``build_agent()`` returns the agent ``execute`` runs, so an eval can
run a task's own agent."""
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def test_execute_runs_the_agent_build_agent_returns(self, mock_hook_cls,
make_mock_run_result):
+ create_agent = mock_hook_cls.get_hook.return_value.create_agent
+ mock_agent = MagicMock(spec=["run_sync", "instrument"])
+ mock_agent.run_sync.return_value = make_mock_run_result("ok")
+ create_agent.return_value = mock_agent
+ op = LLMOperator(
+ task_id="t",
+ prompt="p",
+ llm_conn_id="my_llm",
+ system_prompt="Be brief.",
+ model_settings={"temperature": 0},
+ output_retries=2,
+ )
+ expected = call(
+ output_type=str, instructions="Be brief.",
model_settings={"temperature": 0}, retries=2
+ )
+
+ assert op.build_agent() is mock_agent
+ op.execute(context=_make_context(ti_id="ti-1"))
+
+ assert create_agent.call_args_list == [expected, expected]
+ mock_agent.run_sync.assert_called_once()
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def
test_build_agent_on_a_task_from_a_dag_passes_a_templated_system_prompt_as_written(
+ self, mock_hook_cls
+ ):
+ with DAG("d", schedule=None) as dag:
+ LLMOperator(
+ task_id="summarize", prompt="p", llm_conn_id="my_llm",
system_prompt="Summarize {{ ds }}."
+ )
+
+ dag.get_task("summarize").build_agent()
+
+ create_agent = mock_hook_cls.get_hook.return_value.create_agent
+ assert create_agent.call_args.kwargs["instructions"] == "Summarize {{
ds }}."
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def test_build_agent_on_a_decorated_task(self, mock_hook_cls):
+ with DAG("d", schedule=None) as dag:
+
+ @task.llm(llm_conn_id="my_llm", system_prompt="Be brief.",
output_type=Summary)
+ def summarize() -> str:
+ return "p"
+
+ summarize()
+
+ dag.get_task("summarize").build_agent()
+
+ kwargs =
mock_hook_cls.get_hook.return_value.create_agent.call_args.kwargs
+ assert kwargs["output_type"] is Summary
+ assert kwargs["instructions"] == "Be brief."
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def
test_branch_operator_builds_an_agent_that_picks_a_downstream_task(self,
mock_hook_cls):
+ op = LLMBranchOperator(task_id="route", prompt="p",
llm_conn_id="my_llm", system_prompt="Route it.")
+ op.downstream_task_ids = {"billing", "support"}
+
+ op.build_agent()
+
+ kwargs =
mock_hook_cls.get_hook.return_value.create_agent.call_args.kwargs
+ assert set(TypeAdapter(kwargs["output_type"]).json_schema()["enum"])
== {"billing", "support"}
+ assert kwargs["instructions"] == "Route it."
+
+ def
test_branch_operator_without_downstream_tasks_cannot_build_an_agent(self):
+ op = LLMBranchOperator(task_id="route", prompt="p",
llm_conn_id="my_llm")
+
+ with pytest.raises(ValueError, match="has no downstream tasks"):
+ op.build_agent()
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def
test_file_analysis_operator_builds_an_agent_with_its_system_prompt(self,
mock_hook_cls):
+ op = LLMFileAnalysisOperator(
+ task_id="t", prompt="p", llm_conn_id="my_llm",
file_path="/tmp/app.log", output_type=Summary
+ )
+
+ op.build_agent()
+
+ kwargs =
mock_hook_cls.get_hook.return_value.create_agent.call_args.kwargs
+ assert kwargs["output_type"] is Summary
+ assert kwargs["instructions"] == op._build_system_prompt()
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def
test_sql_operator_builds_an_agent_with_the_schema_in_its_system_prompt(self,
mock_hook_cls):
+ op = LLMSQLQueryOperator(task_id="t", prompt="p", llm_conn_id="my_llm")
+ schema = "Table: orders\nColumns: id INTEGER"
+
+ with patch.object(LLMSQLQueryOperator, "_get_schema_context",
return_value=schema):
Review Comment:
Can this `patch.object` (and the one on line 695) take `autospec=True` like
the other patches in this class?
##########
providers/common/ai/tests/unit/common/ai/operators/test_llm.py:
##########
@@ -577,6 +580,126 @@ def _make_context(ti_id=None):
return MagicMock(**{"__getitem__": lambda self, key: {"task_instance":
ti}[key]})
+class TestLLMOperatorBuildAgent:
+ """``build_agent()`` returns the agent ``execute`` runs, so an eval can
run a task's own agent."""
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def test_execute_runs_the_agent_build_agent_returns(self, mock_hook_cls,
make_mock_run_result):
+ create_agent = mock_hook_cls.get_hook.return_value.create_agent
+ mock_agent = MagicMock(spec=["run_sync", "instrument"])
+ mock_agent.run_sync.return_value = make_mock_run_result("ok")
+ create_agent.return_value = mock_agent
+ op = LLMOperator(
+ task_id="t",
+ prompt="p",
+ llm_conn_id="my_llm",
+ system_prompt="Be brief.",
+ model_settings={"temperature": 0},
+ output_retries=2,
+ )
+ expected = call(
+ output_type=str, instructions="Be brief.",
model_settings={"temperature": 0}, retries=2
+ )
+
+ assert op.build_agent() is mock_agent
+ op.execute(context=_make_context(ti_id="ti-1"))
+
+ assert create_agent.call_args_list == [expected, expected]
+ mock_agent.run_sync.assert_called_once()
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def
test_build_agent_on_a_task_from_a_dag_passes_a_templated_system_prompt_as_written(
+ self, mock_hook_cls
+ ):
+ with DAG("d", schedule=None) as dag:
+ LLMOperator(
+ task_id="summarize", prompt="p", llm_conn_id="my_llm",
system_prompt="Summarize {{ ds }}."
+ )
+
+ dag.get_task("summarize").build_agent()
+
+ create_agent = mock_hook_cls.get_hook.return_value.create_agent
+ assert create_agent.call_args.kwargs["instructions"] == "Summarize {{
ds }}."
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def test_build_agent_on_a_decorated_task(self, mock_hook_cls):
+ with DAG("d", schedule=None) as dag:
+
+ @task.llm(llm_conn_id="my_llm", system_prompt="Be brief.",
output_type=Summary)
+ def summarize() -> str:
+ return "p"
+
+ summarize()
+
+ dag.get_task("summarize").build_agent()
+
+ kwargs =
mock_hook_cls.get_hook.return_value.create_agent.call_args.kwargs
+ assert kwargs["output_type"] is Summary
+ assert kwargs["instructions"] == "Be brief."
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def
test_branch_operator_builds_an_agent_that_picks_a_downstream_task(self,
mock_hook_cls):
+ op = LLMBranchOperator(task_id="route", prompt="p",
llm_conn_id="my_llm", system_prompt="Route it.")
+ op.downstream_task_ids = {"billing", "support"}
+
+ op.build_agent()
+
+ kwargs =
mock_hook_cls.get_hook.return_value.create_agent.call_args.kwargs
+ assert set(TypeAdapter(kwargs["output_type"]).json_schema()["enum"])
== {"billing", "support"}
+ assert kwargs["instructions"] == "Route it."
+
+ def
test_branch_operator_without_downstream_tasks_cannot_build_an_agent(self):
+ op = LLMBranchOperator(task_id="route", prompt="p",
llm_conn_id="my_llm")
+
+ with pytest.raises(ValueError, match="has no downstream tasks"):
+ op.build_agent()
+
+ @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook",
autospec=True)
+ def
test_file_analysis_operator_builds_an_agent_with_its_system_prompt(self,
mock_hook_cls):
Review Comment:
Unlike the `LLMOperator` test above, nothing checks that the file-analysis
`execute()` runs the agent `build_agent()` returns. The existing file-analysis
tests only assert the `create_agent` kwarg names, so `execute()` building its
own agent with different instructions stays green. Could this also call
`execute()` and compare both `create_agent` calls?
##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
The test above scripts the model's replies, so it can't tell you whether a
changed
prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model
and
-a library such as `pydantic-evals
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+ # evals/eval_orders_report.py
+ from pydantic_evals import Case, Dataset
+ from pydantic_evals.evaluators import Contains
+
+ from orders_report import orders_report # dags/orders_report.py from above
+
+ agent = orders_report().get_task("orders_question").build_agent()
+
+ dataset = Dataset(
+ name="orders_question",
+ cases=[Case(inputs="How many orders are there?",
evaluators=[Contains("3")])],
+ )
+ report = dataset.evaluate_sync(lambda question:
agent.run_sync(question).output)
+ report.print()
+
+``build_agent()`` runs outside a task, so templates aren't rendered and a
templated
+``system_prompt`` reaches the agent as written. ``@task.llm``, ``LLMOperator``
and
+the other LLM operators have it too; for ``LLMSQLQueryOperator`` and
Review Comment:
`LLMBatchOperator` is one of the LLM operators but has no `build_agent()`,
since it subclasses `BaseOperator`. Can this name the operators that do?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]