kaxil opened a new pull request, #76862: URL: https://github.com/apache/airflow/pull/76862
When a tool raises an ordinary exception because of the arguments the model chose (a `ValueError` from a user tool, a SQL error from a bad query), a durable agent task fails the same way on every retry. `DurableRun.fail` makes the retry run live from the step that raised, but the model response that chose those arguments sits at an earlier position, so the retry replays it, calls the tool with the same arguments, and raises again until the task is out of retries. A non-durable retry would have asked the model again. The journal has worked this way since `AirflowDurability` was added in #74385; #74720 added the "ask the model again" path, but only for a run that fails on the model's answers without a step raising (`UnexpectedModelBehavior` in pydantic-ai). The run's entry in the journal now remembers the tool call that failed the run and on how many attempts in a row it did. The call is matched on its position, step name and fingerprint, and the fingerprint covers the arguments and the model's call id. The first retry still replays the answer and runs the same call again, so a dropped connection costs one retry and no model call. If the same call fails the run again, the next retry asks the model again from the request that chose it, through the same `model_at_fault` path #74720 added. The count then starts over, so a call from the model's new answer also gets one retry with the same arguments before the model is asked again. The attempt that fails the second time logs: ``` Durable: the same tool call failed on consecutive attempts; a retry asks the model again from the request that chose it position=1 step=slogan_agent__function_toolset__marketing.call_tool:publish_slogan attempts=2 ``` Old journal entries have no `tool_failure` key and start the count at zero; older code ignores the key. ## Design rationale **Why in the journal, not the pydantic-ai capability.** `DurableRun.fail` already decides where a retry runs live for every adapter. Counting there means the OpenAI Agents adapter, which calls the same `step.fail` / `run.fail`, gets the rule too; its docs and the `run_durably` docstring are updated. Its unit tests need the `agents` package and were skipped in my local runs, so for that adapter the journal-level tests are the only evidence. **Only tool steps count.** When a model request itself fails twice, say on a rate limit, there is no earlier answer to replace, so retries keep running it live as before. Another capability's `@durable_operation` isn't counted either, because the model never picked its inputs. **A failure inside a nested agent is counted once, by that agent.** When a tool runs another agent and that agent's tool fails, the same exception fails both runs. Only the inner run counts it, so only the inner model is asked again and the outer answer, and any sibling calls it made, keep replaying. The journal tells the two apart by run key: the outer run skips an error that a run numbered under one of its raising steps has already taken as its own, because its tool call failed or its model is asked again. If the nested agent fails anywhere else, in its own model request or on a usage limit, the outer tool call that ran it is counted like any other failing tool: the arguments the outer model passed are all the outer run can change. A fallback agent started inside an `except` for an earlier run's failure still counts its own failures, even though that earlier error is in its exception context. **Asking again does not show the model the exception.** The model gets the conversation it had before, so it can pick the same arguments again, and each such answer costs two more task retries. Replaying the second failure as a `ModelRetry` would show it the error. I didn't do that: an outage would then reach the model as a complaint about its arguments, and today the journal records only what a tool raises on purpose (`ModelRetry`, `ToolFailed`, the deferral signals) and runs every other failed call again. The docs say this and point to raising `ModelRetry` from the tool when the model should see the error. ## Checked end to end A local Airflow (main) with this branch's code ran an `AgentOperator(durable=True, retries=4)` against a real model (`openai:gpt-5.4-mini`) whose only tool rejects whatever slogan it receives first: | Try | Model | `publish_slogan` | Durable lines in the task log | |---|---|---|---| | 1 | asked | `'Sun-charged light for every campsite'`, raises | `replayed 0 steps (0 model, 0 tool, 0 other), recorded 1 new steps (1 model, 0 tool, 0 other)` | | 2 | replayed | same slogan, raises | `the same tool call failed on consecutive attempts` (`attempts=2`), then `replayed 1 steps (1 model, 0 tool, 0 other), recorded 0 new steps (0 model, 0 tool, 0 other)` | | 3 | asked again | `'Bright camps, zero wires, endless adventure'`, succeeds | `the run took a different path from the previous attempt` at `step=slogan_agent__model.request` (`reason=the model's answers from this step on failed the previous attempt, so it is asked again`), then `replayed 0 steps (0 model, 0 tool, 0 other), recorded 3 new steps (2 model, 1 tool, 0 other)` |   On the base commit, 9 of the 11 new tests fail: attempt 3 replays the first answer and raises again. The other two pass on both: they check that the model is not asked again when a model request fails twice, or when no model request chose the failing call. ## Known issues - The count only moves when a run fails. A run that succeeds and is then retried because of the task's post-run work, or an attempt killed before it can record its failure, leaves the previous count in place, so the model can be asked again one attempt early. - When two parallel tool calls from one answer both fail, pydantic-ai fails the run with whichever finishes first and cancels the other. If they alternate between attempts the count never reaches two, and retries behave as before this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
