kaxil opened a new pull request, #74720: URL: https://github.com/apache/airflow/pull/74720
With `durable=True`, an agent run that pydantic-ai fails with `UnexpectedModelBehavior` could never succeed on a retry. That error is how pydantic-ai gives up when the model keeps giving answers a tool rejects with `ModelRetry` (`SQLToolset` does this for a query outside `allowed_tables` or one it cannot parse), or output that fails validation, past the retry limit. The rejected answers are recorded in the journal like any other step, so every Airflow retry replayed them into the identical error without a single live model call. Without `durable`, the same retry would have asked the model again. The journal decides where a retry stops replaying by matching the run's exception against the steps that raised it. This error is raised by pydantic-ai after the last recorded step, so nothing matched and the retry trusted everything. `DurableRun.fail` now takes `model_at_fault`, which `AirflowDurability` sets for `UnexpectedModelBehavior`: when no step raised the error, the retry replays the steps before the model's last request and asks the model again from there. Before: the retry replays all four recorded steps and fails the same way.  After: the retry replays the first request and its rejected query, asks the model again at position 2, and succeeds.  Both runs are the same Dag in `airflow standalone`: a plain `@task` running a pydantic-ai agent with `AirflowDurability`, the real `SQLToolset` against SQLite, and a scripted model that queries a table outside `allowed_tables` on try 1 and the right table whenever it is asked live afterwards. ## Design rationale **Why not treat every unattributed failure this way.** A cancelled or killed run (`on_kill`, `execution_timeout`) also reaches `fail` without a step to blame, and there replaying everything recorded and running the interrupted step live is right. Re-asking the model would pay for a request the run already has. `UsageLimitExceeded` is deterministic, so a live re-ask would only fail again at a cost. Only `UnexpectedModelBehavior` says the model's own answers are the problem, so the journal takes the flag from the adapter rather than guessing, and stays framework-neutral. **Why the model's last request rather than the first rejected one.** The earlier rejected answers stay in the history and count against pydantic-ai's retry limit, so each Airflow retry gives the model one more answer with the rejections it already got as context. Rewinding further would restore the full budget but also re-run every tool call those earlier answers made. **A step that raised still decides.** When the error propagates out of a step (an agent a tool runs giving up on its own model), that step's position wins and the outer model's answer keeps replaying, as before this change. **A previous attempt's divergence point now holds until an attempt records over it.** Otherwise, if the re-asked request itself fails (a 429), the next retry would replay the older attempt's rejected answer at that position, since the request is identical and its fingerprint matches. The same rule closes the general case of a step that raised at the point an earlier attempt diverged. ## Gotchas Tool calls the model's last answer made run again if the model asks for them again, including calls that succeeded alongside the rejected one. `durable_execution.rst` now says so, both where the retry behaviour is described and in the side-effects section. --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
