This is an automated email from the ASF dual-hosted git repository.
amoghrajesh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new b099814d51f Document how clearing tasks works with task state store on
durable operators (#71358)
b099814d51f is described below
commit b099814d51f208563004c7fe70eaee72d804a516
Author: Amogh Desai <[email protected]>
AuthorDate: Wed Aug 12 09:46:59 2026 +0530
Document how clearing tasks works with task state store on durable
operators (#71358)
---
providers/amazon/docs/operators/redshift/redshift_data.rst | 10 ++++++++++
providers/apache/spark/docs/operators.rst | 10 ++++++++++
providers/databricks/docs/operators/run_now.rst | 10 ++++++++++
providers/databricks/docs/operators/submit_run.rst | 10 ++++++++++
providers/snowflake/docs/operators/snowflake.rst | 10 ++++++++++
5 files changed, 50 insertions(+)
diff --git a/providers/amazon/docs/operators/redshift/redshift_data.rst
b/providers/amazon/docs/operators/redshift/redshift_data.rst
index c9e64aeb662..35d3bf19a07 100644
--- a/providers/amazon/docs/operators/redshift/redshift_data.rst
+++ b/providers/amazon/docs/operators/redshift/redshift_data.rst
@@ -99,6 +99,16 @@ the statement id won't be there for the next retry, and the
operator will submit
instead of reconnecting. Avoid running cleanup on a schedule shorter than your
longest
``retry_delay``.
+Clearing a task is treated the same as a retry, which matters specifically for
a task whose
+statement already succeeded: clearing does not delete the stored statement id,
so the next attempt
+reads it back and returns immediately without submitting the SQL again. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
+``[state_store] clear_on_success`` setting that restores "clearing always
resubmits."
+
+This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
+polling synchronously can cancel the statement via ``on_kill`` before the next
attempt gets a
+chance to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks`
for why.
+
To opt out and always submit fresh SQL on retry, set ``durable=False``:
.. code-block:: python
diff --git a/providers/apache/spark/docs/operators.rst
b/providers/apache/spark/docs/operators.rst
index 9705e505b57..e0060f6db02 100644
--- a/providers/apache/spark/docs/operators.rst
+++ b/providers/apache/spark/docs/operators.rst
@@ -215,6 +215,16 @@ See :doc:`connections/spark-submit` for how to configure
these fields.
Crash recovery in cluster mode requires Airflow 3.3+ (``task_state_store``
support). On earlier
versions the operator falls back to the previous behavior of always
submitting fresh.
+Clearing a task is treated the same as a retry, which matters specifically for
a task whose driver
+already succeeded: clearing does not delete the stored driver ID, so the next
attempt reads it
+back and returns immediately without resubmitting. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
+``[state_store] clear_on_success`` setting that restores "clearing always
resubmits."
+
+This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
+polling synchronously can cancel the driver via ``on_kill`` before the next
attempt gets a chance
+to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for
why.
+
Tracking driver status via Kubernetes API
""""""""""""""""""""""""""""""""""""""""""
diff --git a/providers/databricks/docs/operators/run_now.rst
b/providers/databricks/docs/operators/run_now.rst
index 156049444e6..ff34ccf6242 100644
--- a/providers/databricks/docs/operators/run_now.rst
+++ b/providers/databricks/docs/operators/run_now.rst
@@ -112,6 +112,16 @@ when someone runs ``airflow state-store clean``. If a
task's ``retry_delay`` is
run id won't be there for the next retry, and the operator will trigger a
fresh run instead of
reconnecting. Avoid running cleanup on a schedule shorter than your longest
``retry_delay``.
+Clearing a task is treated the same as a retry, which matters specifically for
a task whose run
+already succeeded: clearing does not delete the stored run id, so the next
attempt reads it back
+and returns immediately without triggering a new run. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
+``[state_store] clear_on_success`` setting that restores "clearing always
resubmits."
+
+This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
+polling synchronously can cancel the run via ``on_kill`` before the next
attempt gets a chance to
+reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.
+
To opt out and always trigger a fresh run on retry, set ``durable=False``:
.. code-block:: python
diff --git a/providers/databricks/docs/operators/submit_run.rst
b/providers/databricks/docs/operators/submit_run.rst
index f3afd104619..76e1e15fcb8 100644
--- a/providers/databricks/docs/operators/submit_run.rst
+++ b/providers/databricks/docs/operators/submit_run.rst
@@ -190,6 +190,16 @@ when someone runs ``airflow state-store clean``. If a
task's ``retry_delay`` is
run id won't be there for the next retry, and the operator will submit a fresh
run instead of
reconnecting. Avoid running cleanup on a schedule shorter than your longest
``retry_delay``.
+Clearing a task is treated the same as a retry, which matters specifically for
a task whose run
+already succeeded: clearing does not delete the stored run id, so the next
attempt reads it back
+and returns immediately without submitting a new run. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
+``[state_store] clear_on_success`` setting that restores "clearing always
resubmits."
+
+This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
+polling synchronously can cancel the run via ``on_kill`` before the next
attempt gets a chance to
+reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.
+
To opt out and always submit a fresh run on retry, set ``durable=False``:
.. code-block:: python
diff --git a/providers/snowflake/docs/operators/snowflake.rst
b/providers/snowflake/docs/operators/snowflake.rst
index 6f12197cd12..113b1832b5b 100644
--- a/providers/snowflake/docs/operators/snowflake.rst
+++ b/providers/snowflake/docs/operators/snowflake.rst
@@ -189,6 +189,16 @@ between, the handles won't be there for the next retry,
and the operator will su
fresh instead of reconnecting. Avoid running cleanup on a schedule shorter
than your longest
``retry_delay``.
+Clearing a task is treated the same as a retry, which matters specifically for
a task whose
+statements already succeeded: clearing does not delete the stored handles, so
the next attempt
+reads them back and returns immediately without submitting the SQL again. See
+:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
+``[state_store] clear_on_success`` setting that restores "clearing always
resubmits."
+
+This is most reliable for deferred tasks (``deferrable=True``); clearing a
task that's actively
+polling synchronously can cancel the statements via ``on_kill`` before the
next attempt gets a
+chance to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks`
for why.
+
To opt out and always submit fresh SQL on retry, set ``durable=False``:
.. code-block:: python
