This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new f5f431378f1 Document effect of state-store cleanup for operators with
durable execution (#70721)
f5f431378f1 is described below
commit f5f431378f19eef122ca220361c4f009272981c1
Author: Amogh Desai <[email protected]>
AuthorDate: Thu Jul 30 15:10:38 2026 +0530
Document effect of state-store cleanup for operators with durable execution
(#70721)
---
providers/amazon/docs/operators/redshift/redshift_data.rst | 7 +++++++
providers/databricks/docs/operators/run_now.rst | 6 ++++++
providers/databricks/docs/operators/submit_run.rst | 6 ++++++
providers/google/docs/operators/cloud/bigquery.rst | 6 ++++++
providers/snowflake/docs/operators/snowflake.rst | 7 +++++++
5 files changed, 32 insertions(+)
diff --git a/providers/amazon/docs/operators/redshift/redshift_data.rst
b/providers/amazon/docs/operators/redshift/redshift_data.rst
index e48b09ce60d..c9e64aeb662 100644
--- a/providers/amazon/docs/operators/redshift/redshift_data.rst
+++ b/providers/amazon/docs/operators/redshift/redshift_data.rst
@@ -92,6 +92,13 @@ earlier Airflow versions the flag is a no-op and the
operator always submits fre
exactly as before. If the task state store is unavailable at runtime, the
operator logs that
crash recovery is disabled and behaves the same way.
+Like the persisted state itself, the stored statement id isn't deleted
automatically, that only
+happens when someone runs ``airflow state-store clean``. If a task's
``retry_delay`` is longer
+than ``[state_store] default_retention_days`` (30 days by default) and cleanup
runs in between,
+the statement id won't be there for the next retry, and the operator will
submit the SQL fresh
+instead of reconnecting. Avoid running cleanup on a schedule shorter than your
longest
+``retry_delay``.
+
To opt out and always submit fresh SQL on retry, set ``durable=False``:
.. code-block:: python
diff --git a/providers/databricks/docs/operators/run_now.rst
b/providers/databricks/docs/operators/run_now.rst
index 78e5a1788aa..00ee6ed0fec 100644
--- a/providers/databricks/docs/operators/run_now.rst
+++ b/providers/databricks/docs/operators/run_now.rst
@@ -99,6 +99,12 @@ Airflow versions the flag is a no-op and the operator always
triggers a fresh ru
exactly as before. If the task state store is unavailable at runtime, the
operator logs that crash
recovery is disabled and behaves the same way.
+Like the persisted state itself, the stored run id isn't deleted
automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay``
is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs
in between, the
+run id won't be there for the next retry, and the operator will trigger a
fresh run instead of
+reconnecting. Avoid running cleanup on a schedule shorter than your longest
``retry_delay``.
+
To opt out and always trigger a fresh run on retry, set ``durable=False``:
.. code-block:: python
diff --git a/providers/databricks/docs/operators/submit_run.rst
b/providers/databricks/docs/operators/submit_run.rst
index 808f7534995..f3afd104619 100644
--- a/providers/databricks/docs/operators/submit_run.rst
+++ b/providers/databricks/docs/operators/submit_run.rst
@@ -184,6 +184,12 @@ earlier Airflow versions the flag is a no-op and the
operator always submits a f
run on retry, exactly as before. If the task state store is unavailable at
runtime, the
operator logs that crash recovery is disabled and behaves the same way.
+Like the persisted state itself, the stored run id isn't deleted
automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay``
is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs
in between, the
+run id won't be there for the next retry, and the operator will submit a fresh
run instead of
+reconnecting. Avoid running cleanup on a schedule shorter than your longest
``retry_delay``.
+
To opt out and always submit a fresh run on retry, set ``durable=False``:
.. code-block:: python
diff --git a/providers/google/docs/operators/cloud/bigquery.rst
b/providers/google/docs/operators/cloud/bigquery.rst
index 9f46ef351fa..6f42b4e841a 100644
--- a/providers/google/docs/operators/cloud/bigquery.rst
+++ b/providers/google/docs/operators/cloud/bigquery.rst
@@ -408,6 +408,12 @@ exactly as before -- including the pre-existing
``reattach_states``/``Conflict``
unchanged. If the task state store is unavailable at runtime, the operator
logs that crash
recovery is disabled and behaves the same way.
+Like the persisted state itself, the stored job id isn't deleted
automatically, that only happens
+when someone runs ``airflow state-store clean``. If a task's ``retry_delay``
is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs
in between, the
+job id won't be there for the next retry, and the operator will submit a fresh
job instead of
+reconnecting. Avoid running cleanup on a schedule shorter than your longest
``retry_delay``.
+
To opt out and always submit a fresh job on retry, set ``durable=False``:
.. code-block:: python
diff --git a/providers/snowflake/docs/operators/snowflake.rst
b/providers/snowflake/docs/operators/snowflake.rst
index 5bb2beb8dac..6f12197cd12 100644
--- a/providers/snowflake/docs/operators/snowflake.rst
+++ b/providers/snowflake/docs/operators/snowflake.rst
@@ -182,6 +182,13 @@ earlier Airflow versions the flag is a no-op and the
operator always submits fre
exactly as before. If the task state store is unavailable at runtime, the
operator logs that crash
recovery is disabled and behaves the same way.
+Like the persisted state itself, the stored statement handles aren't deleted
automatically, that
+only happens when someone runs ``airflow state-store clean``. If a task's
``retry_delay`` is
+longer than ``[state_store] default_retention_days`` (30 days by default) and
cleanup runs in
+between, the handles won't be there for the next retry, and the operator will
submit the SQL
+fresh instead of reconnecting. Avoid running cleanup on a schedule shorter
than your longest
+``retry_delay``.
+
To opt out and always submit fresh SQL on retry, set ``durable=False``:
.. code-block:: python
