amoghrajesh commented on code in PR #76968:
URL: https://github.com/apache/airflow/pull/76968#discussion_r4585503075


##########
providers/databricks/docs/operators/submit_run.rst:
##########
@@ -211,11 +211,30 @@ when someone runs ``airflow state-store clean``. If a 
task's ``retry_delay`` is
 run id won't be there for the next retry, and the operator will submit a fresh 
run instead of
 reconnecting. Avoid running cleanup on a schedule shorter than your longest 
``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose run
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without submitting a new run. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. What the next attempt does depends on the state 
of the Databricks run
+at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits a new run. The first one keeps running.
+     - Reconnects to the running run.
+   * - Succeeded
+     - Submits a new run.
+     - Returns right away without submitting another one.
+   * - Failed or canceled
+     - Submits a new run.
+     - Submits a new run.
+
+Keep task state when a run is still running and nothing about the code or 
inputs changed, so you
+don't pay for two. See :doc:`apache-airflow:core-concepts/resumable-tasks` for 
the ways of clearing.

Review Comment:
   Applied.
   



##########
providers/databricks/docs/operators/run_now.rst:
##########
@@ -143,11 +143,30 @@ when someone runs ``airflow state-store clean``. If a 
task's ``retry_delay`` is
 run id won't be there for the next retry, and the operator will trigger a 
fresh run instead of
 reconnecting. Avoid running cleanup on a schedule shorter than your longest 
``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose run
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without triggering a new run. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. What the next attempt does depends on the state 
of the Databricks run
+at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Triggers a new job run. The first one keeps running.

Review Comment:
   The row now reads "Still running (the task had already failed, or its worker 
died)" on all five pages. Note: I also replaced the "most reliable for deferred 
tasks" line, since the Databricks, Snowflake and Redshift Data triggers cancel 
on clear too. On Spark I dropped it, as Spark has no deferrable mode.



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -203,11 +203,30 @@ id won't be there for the next retry, and the operator 
falls back to the XCom/sc
 above rather than reconnecting via task state store. Avoid running cleanup on 
a schedule shorter
 than your longest ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose job
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without submitting anything to Glue. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. What the next attempt does depends on the state 
of the Glue job run at
+that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Job run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits a new job run. The first one keeps running.

Review Comment:
   The table now covers synchronous mode only, with a line after it for 
`deferrable=True`: a clear reconnects to the running job run whether or not 
task state was kept.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to