kaxil opened a new pull request, #74735:
URL: https://github.com/apache/airflow/pull/74735
`AgentOperator(durable=True)` refused a `CodeMode` capability with
`ValueError("durable=True cannot be used with a CodeMode capability.")`. This
PR removes that error, so an agent that uses code mode can retry from its last
completed step like any other durable agent.
## How a retry replays code mode
`run_code` is not a journal step of its own. On retry each snippet runs
again in Monty, and every tool call the snippet makes replays from the journal
one by one, the same way pydantic-ai-harness handles Temporal. The REPL keeps
state between snippets, so a recorded `run_code` result could not stand in for
re-running it.
That alone did not work, because of the fingerprint. `CodeMode` puts a fresh
`ToolReturnPart` for every nested call in the `run_code` return's
`metadata["tool_returns"]`, and each one carries a new `timestamp`. Those
timestamps were hashed into the next model request's fingerprint, so a retry
diverged at the first model request after any `run_code` call and ran the rest
of the run live.
## Design rationale
**The fingerprint drops only those timestamps.** `_strip_part` removes
`timestamp` from each nested return, and only on a tool return whose metadata
carries `code_mode: True`. That is the shape the harness writes and documents.
The rest of `metadata` stays in the hash, because pydantic-ai keeps routing
state there (`FallbackModel`'s continuation pin under `__pydantic_ai__`), and
the existing test that metadata changes the fingerprint still passes. The
nested calls' tool names, arguments, ids and results are all still hashed.
**A reordered `asyncio.gather` runs live; it never replays the wrong
result.** Code mode numbers a snippet's nested call ids in the order the
snippet makes the calls, and each call's journal fingerprint includes its id
and arguments. When concurrent calls reach the journal in another order on
retry, the first one out of place mismatches, and it and every later step run
live.
**`eager` and `speculate` are not rejected.** They run tool calls while the
model is still writing the snippet, which a positional journal cannot replay.
The harness already turns both off whenever a `BaseDurabilityCapability` is in
durable context, and `AirflowDurability` is one. That check exists in every
harness release that has these options: `eager` from 0.31, `speculate` from
0.33. Refusing the options at parse time would reject a configuration that
already runs safely. The new tests pin it: without durable, eager and
speculation start calls early; with durable, they don't.
**The operator builds no `CodeMode` of its own** (`code_mode=True` went in
#74312), so no stable capability id was needed. A user's `CodeMode()` stays
anonymous, which #74314 and #74423 already made safe by taking `capability_id`
and `deferred_capability_ids` out of the fingerprint.
**The `code-mode` extra floor goes from 0.24.0 to 0.35.0, and the extra
joins the `dev` group.** Harness releases before 0.35.0 accept pydantic-monty
1.x but cannot run code with it. The lowest-dependencies job resolved harness
0.24.0 with monty 1.1.0, and every `run_code` failed. 0.35.0 is the first
release that caps monty below 1.0. The lock moves harness from 0.34.0 to 0.35.0
and nothing else. The `dev` entry makes the new real-`CodeMode` tests run in CI
instead of skipping, as the comment above it already does for `openai` and
`anthropic`.
## Gotchas
The snippet's own code is not recorded, so anything it does outside a tool
call happens again on retry: a write to a `mount` directory, a call to an
`os_access` handler. `durable_execution.rst` now covers this in a new "Code
mode" section.
## Evidence
The end-to-end run used a real `AgentOperator(durable=True,
capabilities=[CodeMode()])` in breeze with `retries=1`, with a scripted
`FunctionModel` standing in for the model. The run's first snippet gathers
`fetch('a')` and `fetch('b')`, and its second calls `fetch('c')`. Try 1 fails
on the third model request with a 503.
Try 2 without the fingerprint change replays one model request and two tool
calls, then diverges at the second model request:
```text
Durable: the run took a different path from the previous attempt; this step
and every step after it run again position=3 step=analyst__model.request
reason=the request differs from the previous attempt's (prompt, model,
settings, tools, arguments or conversation so far)
LIVE model request 2 on try 2
LIVE tool call: fetch(name='c') on try 2
LIVE model request 3 on try 2
Durable: replayed 3 steps (1 model, 2 tool, 0 other), recorded 3 new steps
(2 model, 1 tool, 0 other)
```
Try 2 with it replays everything up to the failure and calls the model only
for the request that failed:
```text
LIVE model request 3 on try 2
run_code returned [['a-data', 'b-data'], 'c-data']
Durable: replayed 5 steps (2 model, 3 tool, 0 other), recorded 1 new steps
(1 model, 0 tool, 0 other)
```
With `fetch('a')` held back on retry until `fetch('b')` has started, the
reordered calls run live and still return the right results:
```text
Durable: the run took a different path from the previous attempt; this step
and every step after it run again position=1
step=analyst__function_toolset__warehouse.call_tool:fetch reason=the request
differs from the previous attempt's (prompt, model, settings, tools, arguments
or conversation so far)
LIVE tool call: fetch(name='b') on try 2
LIVE tool call: fetch(name='a') on try 2
...
run_code returned [['a-data', 'b-data'], 'c-data']
```
`test_a_retry_replays_the_steps_after_a_run_code_call` fails with the
fingerprint change reverted (`{'model': 1, 'tool': 2}` replayed instead of
`{'model': 2, 'tool': 3}`). The new tests pass on harness 0.34.0, 0.35.0 (the
new floor, also under lowest-direct resolution) and 0.54.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
