This is an automated email from the ASF dual-hosted git repository.
amoghrajesh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 22a2fdb930e Link task state store docs in durable execution across
providers (#69851)
22a2fdb930e is described below
commit 22a2fdb930e96ab149dcd1c062dbc66e113cac95
Author: Amogh Desai <[email protected]>
AuthorDate: Tue Jul 14 15:00:39 2026 +0530
Link task state store docs in durable execution across providers (#69851)
---
providers/amazon/docs/operators/redshift/redshift_data.rst | 6 +++---
providers/apache/spark/docs/operators.rst | 6 +++---
providers/databricks/docs/operators/run_now.rst | 7 ++++---
providers/databricks/docs/operators/submit_run.rst | 7 ++++---
providers/google/docs/operators/cloud/bigquery.rst | 7 ++++---
providers/snowflake/docs/operators/snowflake.rst | 7 ++++---
6 files changed, 22 insertions(+), 18 deletions(-)
diff --git a/providers/amazon/docs/operators/redshift/redshift_data.rst
b/providers/amazon/docs/operators/redshift/redshift_data.rst
index 7720e304b76..e48b09ce60d 100644
--- a/providers/amazon/docs/operators/redshift/redshift_data.rst
+++ b/providers/amazon/docs/operators/redshift/redshift_data.rst
@@ -71,9 +71,9 @@ Durable execution
``RedshiftDataOperator`` submits a statement and then polls it to completion
on the worker. By
default the operator runs in a *durable* mode that makes this crash-safe: the
Redshift statement
-id is persisted to task state before polling begins, so if the worker crashes
or is preempted and
-the task is retried, the operator reconnects to the statement that is already
executing in
-Redshift instead of resubmitting the SQL.
+id is persisted to :doc:`task state store
<apache-airflow:core-concepts/task-state-store>` before
+polling begins, so if the worker crashes or is preempted and the task is
retried, the operator
+reconnects to the statement that is already executing in Redshift instead of
resubmitting the SQL.
On retry the operator checks the prior statement's state:
diff --git a/providers/apache/spark/docs/operators.rst
b/providers/apache/spark/docs/operators.rst
index d20c1da5cf7..9705e505b57 100644
--- a/providers/apache/spark/docs/operators.rst
+++ b/providers/apache/spark/docs/operators.rst
@@ -190,9 +190,9 @@ independently on the cluster. If the Airflow worker dies
while the Spark job is
Airflow loses track of it and the behaviour to submit a brand new job would be
wasting
the compute already done or even cause conflicts if the Spark job itself is
not designed to be idempotent.
-Now, the ``SparkSubmitOperator`` solves this by persisting the driver ID to
``task_state_store`` immediately after
-submission. On retry, it reads the ID back and reconnects to the
already-running driver instead of
-resubmitting.
+Now, the ``SparkSubmitOperator`` solves this by persisting the driver ID to
:doc:`task state store
+<apache-airflow:core-concepts/task-state-store>` immediately after submission.
On retry, it reads
+the ID back and reconnects to the already-running driver instead of
resubmitting.
This is the **synchronous path** — the worker holds a slot for the duration of
polling. This is
a crash-safety net for teams running sync operators for log observability, org
constraints, or
diff --git a/providers/databricks/docs/operators/run_now.rst
b/providers/databricks/docs/operators/run_now.rst
index 30f64139b9c..78e5a1788aa 100644
--- a/providers/databricks/docs/operators/run_now.rst
+++ b/providers/databricks/docs/operators/run_now.rst
@@ -79,9 +79,10 @@ Durable execution
``DatabricksRunNowOperator`` triggers a run of an existing job and then polls
it to completion on
the worker. By default the operator runs in a *durable* mode that makes this
crash-safe: the
-Databricks run id is persisted to Airflow's task state store before polling
begins, so if the
-worker crashes or is preempted and the task is retried, the operator
reconnects to the run that is
-already executing on Databricks instead of triggering a duplicate run of the
same job.
+Databricks run id is persisted to :doc:`task state store
+<apache-airflow:core-concepts/task-state-store>` before polling begins, so if
the worker crashes
+or is preempted and the task is retried, the operator reconnects to the run
that is already
+executing on Databricks instead of triggering a duplicate run of the same job.
On retry the operator checks the prior run's state:
diff --git a/providers/databricks/docs/operators/submit_run.rst
b/providers/databricks/docs/operators/submit_run.rst
index f6dbffa5ef1..808f7534995 100644
--- a/providers/databricks/docs/operators/submit_run.rst
+++ b/providers/databricks/docs/operators/submit_run.rst
@@ -165,9 +165,10 @@ Durable execution
``DatabricksSubmitRunOperator`` submits a run and then polls it to completion
on the worker.
By default the operator runs in a *durable* mode that makes the runs
crash-safe: the Databricks
-run id is persisted to task state store before polling begins, so if the
worker crashes or is
-preempted and the task is retried, the operator reconnects to the run that is
already executing
-on Databricks instead of submitting a duplicate.
+run id is persisted to :doc:`task state store
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task
is retried, the
+operator reconnects to the run that is already executing on Databricks instead
of submitting a
+duplicate.
On retry the operator checks the prior run's state:
diff --git a/providers/google/docs/operators/cloud/bigquery.rst
b/providers/google/docs/operators/cloud/bigquery.rst
index 1513a958670..9f46ef351fa 100644
--- a/providers/google/docs/operators/cloud/bigquery.rst
+++ b/providers/google/docs/operators/cloud/bigquery.rst
@@ -374,9 +374,10 @@ Durable execution
``BigQueryInsertJobOperator`` submits a job and then polls it to completion on
the worker. By
default the operator runs in a *durable* mode that makes this crash-safe: the
submitted BigQuery
-job id is persisted to task state before polling begins, so if the worker
crashes or is preempted
-and the task is retried, the operator reconnects to the job that is already
running in BigQuery
-instead of submitting a duplicate.
+job id is persisted to :doc:`task state store
<apache-airflow:core-concepts/task-state-store>`
+before polling begins, so if the worker crashes or is preempted and the task
is retried, the
+operator reconnects to the job that is already running in BigQuery instead of
submitting a
+duplicate.
On retry the operator checks the prior job's state:
diff --git a/providers/snowflake/docs/operators/snowflake.rst
b/providers/snowflake/docs/operators/snowflake.rst
index 85c1df3b2ee..5bb2beb8dac 100644
--- a/providers/snowflake/docs/operators/snowflake.rst
+++ b/providers/snowflake/docs/operators/snowflake.rst
@@ -160,9 +160,10 @@ Durable execution
``SnowflakeSqlApiOperator`` submits one or more SQL statements and then polls
their statement
handles to completion on the worker. By default the operator runs in a
*durable* mode that makes
-this crash-safe: the statement handles are persisted to task state store
before polling begins, so
-if the worker crashes or is preempted and the task is retried, the operator
reconnects to the
-statements that are already executing in Snowflake instead of resubmitting the
SQL.
+this crash-safe: the statement handles are persisted to :doc:`task state store
+<apache-airflow:core-concepts/task-state-store>` before polling begins, so if
the worker crashes
+or is preempted and the task is retried, the operator reconnects to the
statements that are
+already executing in Snowflake instead of resubmitting the SQL.
On retry the operator checks the prior statements' state:
