aarmens702-hub commented on issue #76788:
URL: https://github.com/apache/airflow/issues/76788#issuecomment-6807527288
Thanks, that makes sense, and yes, that would cover it. The eval would then
run the agent the task actually builds, with its connection, instructions,
toolsets and capabilities, instead of a copy that can drift.
Both builders already exist but are private: `AgentOperator._build_agent()`
(which `@task.agent` inherits) and `LLMOperator._create_agent(output_type=...,
instructions=...)`. I'd add a public `build_agent()` with no arguments to both,
returning the agent the task would run. From a test it would look like:
```python
agent = dag.get_task("triage").build_agent()
report = dataset.evaluate_sync(lambda q: agent.run_sync(q).output)
```
Two questions:
1. `system_prompt` is a template field, so outside a task run it isn't
rendered. Should `build_agent()` raise if it still contains Jinja, or leave
that to the caller?
2. Should I update the evals paragraph in `local_development.rst` to use
this?
I'm happy to take this. I'll raise the `averages()` and usage-limit points
with pydantic-evals separately.
Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before
posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
