aaron-y-chen commented on code in PR #76968:
URL: https://github.com/apache/airflow/pull/76968#discussion_r4580713077


##########
providers/amazon/docs/operators/redshift/redshift_data.rst:
##########
@@ -99,11 +99,31 @@ the statement id won't be there for the next retry, and the 
operator will submit
 instead of reconnecting. Avoid running cleanup on a schedule shorter than your 
longest
 ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose
-statement already succeeded: clearing does not delete the stored statement id, 
so the next attempt
-reads it back and returns immediately without submitting the SQL again. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored statement id by default (Airflow 3.4+), and 
so does clearing its
+whole Dag run (Airflow 3.5+). You can choose to keep task state when you 
clear, and the next attempt
+then reads the stored statement id back. What the next attempt does depends on 
the state of the
+statement at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Statement state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits the SQL again. The first statement keeps running.
+     - Reconnects to the running statement.
+   * - Succeeded
+     - Submits the SQL again.
+     - Returns right away without running the SQL again.
+   * - Failed or aborted
+     - Submits the SQL again.
+     - Submits the SQL again.
+
+Keep task state when a statement is still running and nothing about the code 
or inputs changed, so
+you don't pay for two. See :doc:`apache-airflow:core-concepts/resumable-tasks` 
for the ways of
+clearing.

Review Comment:
   IIUC, this could be clearer. WDYT?
   
   ```suggestion
   Keep task state when a statement is still running and nothing about the code 
or inputs changed, so
   you don't pay for two. This assumes the statement survives the clear; 
keeping task state does not
   prevent cancellation via ``on_kill``. See 
:doc:`apache-airflow:core-concepts/resumable-tasks`
   for the ways of clearing.
   ```



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -203,11 +203,30 @@ id won't be there for the next retry, and the operator 
falls back to the XCom/sc
 above rather than reconnecting via task state store. Avoid running cleanup on 
a schedule shorter
 than your longest ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose job
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without submitting anything to Glue. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. What the next attempt does depends on the state 
of the Glue job run at
+that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Job run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits a new job run. The first one keeps running.
+     - Reconnects to the running job run.
+   * - Succeeded
+     - Submits a new job run.
+     - Returns right away without starting another one.
+   * - Failed or stopped
+     - Submits a new job run.
+     - Submits a new job run.
+
+Keep task state when a job run is still running and nothing about the code or 
inputs changed, so you
+don't pay for two. See :doc:`apache-airflow:core-concepts/resumable-tasks` for 
the ways of clearing.

Review Comment:
   Same as above.
   
   ```suggestion
   Keep task state when a statement is still running and nothing about the code 
or inputs changed, so
   you don't pay for two. This assumes the statement survives the clear; 
keeping task state does not
   prevent cancellation via ``on_kill``. See 
:doc:`apache-airflow:core-concepts/resumable-tasks`
   for the ways of clearing.
   ```



##########
providers/apache/spark/docs/operators.rst:
##########
@@ -223,11 +223,30 @@ prerequisite each one needs before the operator can check 
driver status on retry
     immediately) still emits a deprecation warning on every Airflow version 
and maps onto
     ``durable``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose driver
-already succeeded: clearing does not delete the stored driver ID, so the next 
attempt reads it
-back and returns immediately without resubmitting. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored driver ID by default (Airflow 3.4+), and so 
does clearing its
+whole Dag run (Airflow 3.5+). You can choose to keep task state when you 
clear, and the next attempt
+then reads the stored driver ID back. What the next attempt does depends on 
the state of the driver
+at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Driver state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits a new driver. The first one keeps running.
+     - Reconnects to the running driver.
+   * - Succeeded
+     - Submits a new driver.
+     - Returns right away without submitting another one.
+   * - Failed or killed
+     - Submits a new driver.
+     - Submits a new driver.
+
+Keep task state when a driver is still running and nothing about the code or 
inputs changed, so you
+don't pay for two. See :doc:`apache-airflow:core-concepts/resumable-tasks` for 
the ways of clearing.

Review Comment:
   ```suggestion
   Keep task state when a driver is still running and nothing about the code or 
inputs changed, so
   you don't pay for two. This assumes the driver survives the clear; keeping 
task state does not
   prevent cancellation via ``on_kill``. See 
:doc:`apache-airflow:core-concepts/resumable-tasks`
   for the ways of clearing.
   ```



##########
providers/databricks/docs/operators/submit_run.rst:
##########
@@ -211,11 +211,30 @@ when someone runs ``airflow state-store clean``. If a 
task's ``retry_delay`` is
 run id won't be there for the next retry, and the operator will submit a fresh 
run instead of
 reconnecting. Avoid running cleanup on a schedule shorter than your longest 
``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose run
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without submitting a new run. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. What the next attempt does depends on the state 
of the Databricks run
+at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits a new run. The first one keeps running.
+     - Reconnects to the running run.
+   * - Succeeded
+     - Submits a new run.
+     - Returns right away without submitting another one.
+   * - Failed or canceled
+     - Submits a new run.
+     - Submits a new run.
+
+Keep task state when a run is still running and nothing about the code or 
inputs changed, so you
+don't pay for two. See :doc:`apache-airflow:core-concepts/resumable-tasks` for 
the ways of clearing.

Review Comment:
   ```suggestion
   Keep task state when a run is still running and nothing about the code or 
inputs changed, so
   you don't pay for two. This assumes the run survives the clear; keeping task 
state does not
   prevent cancellation via ``on_kill``. See 
:doc:`apache-airflow:core-concepts/resumable-tasks`
   for the ways of clearing.
   ```



##########
providers/snowflake/docs/operators/snowflake.rst:
##########
@@ -189,11 +189,31 @@ between, the handles won't be there for the next retry, 
and the operator will su
 fresh instead of reconnecting. Avoid running cleanup on a schedule shorter 
than your longest
 ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose
-statements already succeeded: clearing does not delete the stored handles, so 
the next attempt
-reads them back and returns immediately without submitting the SQL again. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored statement handles by default (Airflow 
3.4+), and so does clearing
+its whole Dag run (Airflow 3.5+). You can choose to keep task state when you 
clear, and the next
+attempt then reads the stored statement handles back. What the next attempt 
does depends on the
+state of the statements at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Statement state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits the SQL again. The first statements keep running.
+     - Reconnects to the running statements.
+   * - Succeeded
+     - Submits the SQL again.
+     - Returns right away without running the SQL again.
+   * - Failed or aborted
+     - Submits the SQL again.
+     - Submits the SQL again.
+
+Keep task state when statements are still running and nothing about the code 
or inputs changed, so

Review Comment:
   ```suggestion
   Keep task state when statements are still running and nothing about the code 
or inputs changed, so
   you don't pay for two. This assumes the statements survive the clear; 
keeping task state does not
   prevent cancellation via ``on_kill``. See 
:doc:`apache-airflow:core-concepts/resumable-tasks`
   for the ways of clearing.
   ```



##########
providers/databricks/docs/operators/run_now.rst:
##########
@@ -143,11 +143,30 @@ when someone runs ``airflow state-store clean``. If a 
task's ``retry_delay`` is
 run id won't be there for the next retry, and the operator will trigger a 
fresh run instead of
 reconnecting. Avoid running cleanup on a schedule shorter than your longest 
``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose run
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without triggering a new run. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. What the next attempt does depends on the state 
of the Databricks run
+at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Triggers a new job run. The first one keeps running.
+     - Reconnects to the running run.
+   * - Succeeded
+     - Triggers a new job run.
+     - Returns right away without triggering another one.
+   * - Failed or canceled
+     - Triggers a new job run.
+     - Triggers a new job run.
+
+Keep task state when a run is still running and nothing about the code or 
inputs changed, so you
+don't pay for two. See :doc:`apache-airflow:core-concepts/resumable-tasks` for 
the ways of clearing.

Review Comment:
   ```suggestion
   Keep task state when a run is still running and nothing about the code or 
inputs changed, so
   you don't pay for two. This assumes the run survives the clear; keeping task 
state does not
   prevent cancellation via ``on_kill``. See 
:doc:`apache-airflow:core-concepts/resumable-tasks`
   for the ways of clearing.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to