aarmens702-hub opened a new issue, #76788: URL: https://github.com/apache/airflow/issues/76788
### Description Add an experimental `LLMEvalOperator` to the common.ai provider that runs a pydantic-evals dataset inside a Dag and fails the task if the pass rate is below a threshold. It would push a summary to XCom, optionally write the full report to object storage, and live behind a new `evals` extra. ### Use case/motivation `docs/local_development.rst` tells users to run pydantic-evals before shipping a prompt change. That happens outside Airflow, so it can't use Airflow connections or stop a bad prompt from reaching the Dag. A PythonOperator can run evals, but with pydantic-evals 2.66.0 I found three easy mistakes: - `report.averages()` skips crashed cases. With 1 crash in 4 cases it reports 0.83 instead of 0.50. - An `LLMJudge` without a model uses a process-wide default (`openai:gpt-5.2`). - If the usage limit runs out or the task is killed, the remaining cases are recorded as failures. The run looks like a low score instead of an incomplete one. The operator would handle all three. On the dev@ list, kaxil said "Evals will be part of it" ([link](https://lists.apache.org/api/email.lua?id=994jgsysymrlxctw6qzpnc0r3mj6on5g)), but I couldn't find a tracking issue. **Questions before I start** 1. New operator or an option on `LLMOperator`? I suggest a new operator. 2. Should a score below the threshold fail without retries (`AirflowFailException`)? 3. Is it OK to ship the operator first and add `@task.llm_eval` as a follow-up? ### Related issues Separate from the contributor-agent skill evals (#69308) and the data quality provider (#69575). Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before posting ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
