This is an automated email from the ASF dual-hosted git repository. kaxil pushed a commit to branch durable-clear-docs in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d985bc362ea0d6137ffd51e8307be09ea02b9161 Author: Kaxil Naik <[email protected]> AuthorDate: Tue Nov 10 09:58:35 2026 +0000 Correct what clearing a durable agent task replays in common.ai docs The durable execution docs said a clear keeps the agent's journal, so a cleared task replays what its failed attempts recorded. Since Airflow 3.4, clearing a task instance deletes its task state, the journal included, unless keep_task_state is set. Add a "Clearing a task" section that says when a cleared task replays, and point the usage_limits docs at it. --- providers/common/ai/docs/durable_execution.rst | 32 +++++++++++++++++++--- providers/common/ai/docs/operators/agent.rst | 6 ++-- .../airflow/providers/common/ai/operators/agent.py | 11 ++++---- 3 files changed, 37 insertions(+), 12 deletions(-) diff --git a/providers/common/ai/docs/durable_execution.rst b/providers/common/ai/docs/durable_execution.rst index 87bffbbed7a..38f08f4c781 100644 --- a/providers/common/ai/docs/durable_execution.rst +++ b/providers/common/ai/docs/durable_execution.rst @@ -192,7 +192,7 @@ any other exception runs again on retry. A task can run more than one agent, and an agent's tool can run another agent; each agent run is replayed on its own, and one a tool started is replayed under that tool call. The journal is deleted when the task succeeds, including any steps an earlier -attempt recorded beyond where the last one went, so a task cleared later starts fresh. +attempt recorded beyond where the last one went. In a plain ``@task``, which has no such moment, the capability deletes a run's journal as soon as that run succeeds: after a failure, only the run that failed replays, and the agents that ran before it run again. @@ -312,7 +312,8 @@ accounting for a ``run`` window fails with a ``KeyError``. The ``run_id`` XCom, failed attempt or a successful one, and the ``gen_ai.agent.call.id`` span attribute carry the same id; the spans of each attempt still carry that attempt's own ``airflow.task_instance.id`` and ``try_number``. The id is stored with the journal and -deleted with it, so a task cleared after it succeeded starts a new one. +deleted with it, so a rerun after a clear that deleted the journal (see +:ref:`durable-clear`) gets a new one. Usage and limits ---------------- @@ -320,8 +321,31 @@ Usage and limits A replayed step adds nothing to the usage the run counts against ``usage_limits`` or reports in the ``usage`` XCom: not its request, tokens, cost or tool calls, nor the usage a replayed ``@durable_operation`` brings back. Each attempt counts only the calls -it actually makes. This holds after clearing a failed task instance too: the clear -starts a fresh budget but keeps the journal, and what the rerun replays from it is free. +it actually makes. Clearing a finished task instance starts a fresh budget; if the +rerun replays from the journal (see :ref:`durable-clear`), the replayed steps cost nothing. + +.. _durable-clear: + +Clearing a task +--------------- + +On Airflow 3.4 and later, clearing a task instance (its Clear dialog in the UI, or +``POST /api/v2/dags/{dag_id}/clearTaskInstances``) deletes its task state, the journal +included, so the rerun starts the agent over. This holds for the default task state store +backend; a custom ``[state_store] backend`` keeps its entries. To resume from the journal +instead, tick **Keep task state and resume** in the dialog, or send +``"keep_task_state": true`` in the request. The **Keep task state on clear** UI setting +ticks that box by default. Clearing the whole Dag run from the Clear Run dialog keeps task +state; "Retries resume, clearing starts over" in +:ref:`apache-airflow:concepts-resumable-tasks` lists the other ways of clearing and which +of them keep it. + +Before Airflow 3.4 no clear deletes the journal (the task state store on 3.3, the file +under ``durable_cache_path`` earlier), so a cleared task replays. + +A task that succeeded has no journal left, so clearing it starts over either way. For +``AgentOperator`` and ``@task.agent``, the rerun's ``Durable:`` line in the task log +counts the steps it replayed; a count above zero means it resumed from the journal. Side effects and idempotency ---------------------------- diff --git a/providers/common/ai/docs/operators/agent.rst b/providers/common/ai/docs/operators/agent.rst index 095914469fa..c8b1ea31dd4 100644 --- a/providers/common/ai/docs/operators/agent.rst +++ b/providers/common/ai/docs/operators/agent.rst @@ -440,9 +440,9 @@ Parameters every Airflow version. A step that re-runs live because the conversation changed since the previous attempt is counted like any other live call, and a retry whose cross-attempt total already sits at a limit can still start - when the steps it needs are recorded. Clearing a failed task instance starts - a fresh budget but keeps the journal its attempts left behind, so what the - rerun replays from it is free there too. See :doc:`../durable_execution`. + when the steps it needs are recorded. Clearing a finished task instance + starts a fresh budget; whether its rerun replays from the journal depends on + the Airflow version and how the task was cleared (:ref:`durable-clear`). - ``cache_prompt``: Ask the provider to cache the tool definitions, system prompt and conversation so later requests read them back at a discount. Default ``True``; a no-op for providers that cache on their own. See :ref:`agent-prompt-caching`. diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/agent.py b/providers/common/ai/src/airflow/providers/common/ai/operators/agent.py index 94d86d57e76..05da22c2760 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/agent.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/agent.py @@ -382,11 +382,12 @@ class AgentOperator(CancellableAgentRunMixin, BaseOperator, HITLReviewMixin, Usa a warning). Default ``False``. A replayed step adds nothing to the usage counted against ``usage_limits`` or reported in the ``usage`` XCom -- not its request, tokens, cost, or tool calls -- so every attempt counts only - the model and tool calls it actually makes. This holds the same way after - clearing a failed task instance: it starts a fresh budget but keeps the - journal its attempts left behind, and whatever the rerun replays from it - is free. The agent's ``run_id`` (and its ``conversation_id``, when there is - no message history) stays the first attempt's on every retry. + the model and tool calls it actually makes. Clearing a finished task + instance starts a fresh budget; whether its rerun replays from the journal + depends on the Airflow version and how the task was cleared + (:ref:`durable-clear`). The agent's ``run_id`` (and its + ``conversation_id``, when there is no message history) stays the first + attempt's on every retry. On Airflow >= 3.3 the journal is kept in the AIP-103 task state store, so no extra configuration is needed. On older Airflow versions it is persisted to ObjectStorage and requires ``[common.ai] durable_cache_path`` to be set.
