kaxil commented on issue #76788:
URL: https://github.com/apache/airflow/issues/76788#issuecomment-6807450931
I'd rather not add an operator for this. The case under test is a Python
callable, so `LLMEvalOperator` ends up as a `@task` with a threshold, tied to
pydantic-evals. A judge model from a connection is already
`LLMJudge(model=PydanticAIHook.get_hook("my_llm").get_conn())`, and online
evals can go on `@task.agent` as an `OnlineEvaluation` capability. The
`averages()` and usage-limit points look worth raising upstream in
pydantic-evals. The gap I do see is that the eval runs a copy of the agent, not
the one the task builds. Would a public way to get the agent from `@task.agent`
/ `LLMOperator` cover what you need?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
