kaxil commented on code in PR #76968:
URL: https://github.com/apache/airflow/pull/76968#discussion_r4604088631


##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -203,11 +203,34 @@ id won't be there for the next retry, and the operator 
falls back to the XCom/sc
 above rather than reconnecting via task state store. Avoid running cleanup on 
a schedule shorter
 than your longest ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose job
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without submitting anything to Glue. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. In synchronous mode, what the next attempt does 
depends on the state
+of the Glue job run at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Job run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits a new job run. The first one keeps running.

Review Comment:
   With `concurrent_run_limit` at its default of 1, this doesn't get a second 
run. In sync mode `submit_job` skips the UUID scan once task state is gone, and 
`initialize_job` fails with `ConcurrentRunsExceededException` until the first 
run ends, as the section above says. Can this cell say that, and keep "submits 
a new job run" for a higher limit?



##########
providers/common/ai/docs/durable_execution.rst:
##########
@@ -414,16 +414,13 @@ rerun replays from the journal (see 
:ref:`durable-clear`), the replayed steps co
 Clearing a task
 ---------------
 
-On Airflow 3.4 and later, clearing a task instance (its Clear dialog in the 
UI, or
-``POST /api/v2/dags/{dag_id}/clearTaskInstances``) deletes its task state, the 
journal
-included, so the rerun starts the agent over. This holds for the default task 
state store
-backend; a custom ``[state_store] backend`` keeps its entries. To resume from 
the journal
-instead, tick **Keep task state and resume** in the dialog, or send
-``"keep_task_state": true`` in the request. The **Keep task state on clear** 
UI setting
-ticks that box by default. Clearing the whole Dag run from the Clear Run 
dialog keeps task
-state; "Retries resume, clearing starts over" in
-:ref:`apache-airflow:concepts-resumable-tasks` lists the other ways of 
clearing and which
-of them keep it.
+Clearing a task deletes its task state by default (Airflow 3.4+), and so does 
clearing

Review Comment:
   On 3.4 only the UI Clear dialog and `POST .../clearTaskInstances` (so 
`airflowctl` too) discard task state. `airflow tasks clear` and the downstream 
clear from mark success/failed keep it until #76806 in 3.5. The old text named 
those paths. Can this and the operator pages say UI/REST task clears from 3.4, 
every other clear from 3.5?



##########
providers/amazon/docs/operators/redshift/redshift_data.rst:
##########
@@ -99,15 +99,36 @@ the statement id won't be there for the next retry, and the 
operator will submit
 instead of reconnecting. Avoid running cleanup on a schedule shorter than your 
longest
 ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose
-statement already succeeded: clearing does not delete the stored statement id, 
so the next attempt
-reads it back and returns immediately without submitting the SQL again. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
-
-This is most reliable for deferred tasks (``deferrable=True``); clearing a 
task that's actively
-polling synchronously can cancel the statement via ``on_kill`` before the next 
attempt gets a
-chance to reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` 
for why.
+Clearing a task deletes the stored statement id by default (Airflow 3.4+), and 
so does clearing its
+whole Dag run (Airflow 3.5+). You can choose to keep task state when you 
clear, and the next attempt
+then reads the stored statement id back. What the next attempt does depends on 
the state of the
+statement at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Statement state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running (the task had already failed, or its worker died)
+     - Submits the SQL again. The first statement keeps running.
+     - Reconnects to the running statement.
+   * - Succeeded
+     - Submits the SQL again.
+     - Returns right away without running the SQL again.

Review Comment:
   This only holds in synchronous mode. With `deferrable=True`, `execute` never 
calls `execute_resumable`, so nothing is written to task state and a clear that 
keeps it submits the SQL again. Same for `snowflake.rst` and both Databricks 
pages. Can these tables be scoped to synchronous mode like the Glue one, with a 
line for `deferrable=True`?



##########
providers/common/ai/docs/durable_execution.rst:
##########
@@ -414,16 +414,13 @@ rerun replays from the journal (see 
:ref:`durable-clear`), the replayed steps co
 Clearing a task
 ---------------
 
-On Airflow 3.4 and later, clearing a task instance (its Clear dialog in the 
UI, or
-``POST /api/v2/dags/{dag_id}/clearTaskInstances``) deletes its task state, the 
journal
-included, so the rerun starts the agent over. This holds for the default task 
state store
-backend; a custom ``[state_store] backend`` keeps its entries. To resume from 
the journal
-instead, tick **Keep task state and resume** in the dialog, or send
-``"keep_task_state": true`` in the request. The **Keep task state on clear** 
UI setting
-ticks that box by default. Clearing the whole Dag run from the Clear Run 
dialog keeps task
-state; "Retries resume, clearing starts over" in
-:ref:`apache-airflow:concepts-resumable-tasks` lists the other ways of 
clearing and which
-of them keep it.
+Clearing a task deletes its task state by default (Airflow 3.4+), and so does 
clearing
+its whole Dag run (Airflow 3.5+). The journal goes with it, so the rerun 
starts the agent
+over. This holds for the default task state store backend; a custom
+``[state_store] backend`` keeps its entries. You can choose to keep task state 
when you

Review Comment:
   This lost how to keep it: the **Keep task state and resume** box, 
`"keep_task_state": true` in the request, and the **Keep task state on clear** 
setting. Can those come back? The Glue, Redshift, Spark, Snowflake and HF pages 
also just say "you can choose", where the Databricks pages say to set 
`keep_task_state`.



##########
providers/databricks/docs/operators/run_now.rst:
##########
@@ -142,15 +142,35 @@ stored, so a retry that comes later than that finds 
nothing and triggers a fresh
 reconnecting, whatever schedule ``airflow state-store clean`` runs on. Size 
that setting against
 your longest ``retry_delay``.
 
-Clearing a task discards the stored run id by default, so the next attempt 
triggers a fresh run.
-Set ``keep_task_state`` when clearing to keep it instead, and the attempt is 
then treated the same
-as a retry: for a task whose run already succeeded, it reads the run back and 
returns immediately
-without triggering a new one. 
:doc:`apache-airflow:core-concepts/resumable-tasks` lists which
-clear paths discard the id and describes the ``[state_store] 
clear_on_success`` setting.
-
-This is most reliable for deferred tasks (``deferrable=True``); clearing a 
task that's actively
-polling synchronously can cancel the run via ``on_kill`` before the next 
attempt gets a chance to
-reconnect -- see :doc:`apache-airflow:core-concepts/resumable-tasks` for why.
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). Set ``keep_task_state`` when clearing to keep it, and 
the next attempt then
+reads the stored run id back. What the next attempt does depends on the state 
of the Databricks run
+at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running (the task had already failed, or its worker died)
+     - Triggers a new job run. The first one keeps running.

Review Comment:
   With `cancel_previous_runs=True`, `submit_job` cancels every active run of 
the job first, so the first one doesn't keep running. Worth an "unless" here?



##########
providers/amazon/docs/operators/glue.rst:
##########
@@ -203,11 +203,34 @@ id won't be there for the next retry, and the operator 
falls back to the XCom/sc
 above rather than reconnecting via task state store. Avoid running cleanup on 
a schedule shorter
 than your longest ``retry_delay``.
 
-Clearing a task is treated the same as a retry, which matters specifically for 
a task whose job
-already succeeded: clearing does not delete the stored run id, so the next 
attempt reads it back
-and returns immediately without submitting anything to Glue. See
-:doc:`apache-airflow:core-concepts/resumable-tasks` for why, and for the
-``[state_store] clear_on_success`` setting that restores "clearing always 
resubmits."
+Clearing a task deletes the stored run id by default (Airflow 3.4+), and so 
does clearing its whole
+Dag run (Airflow 3.5+). You can choose to keep task state when you clear, and 
the next attempt then
+reads the stored run id back. In synchronous mode, what the next attempt does 
depends on the state
+of the Glue job run at that point:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Job run state
+     - Default clear
+     - Clear that keeps task state
+   * - Still running
+     - Submits a new job run. The first one keeps running.
+     - Reconnects to the running job run.
+   * - Succeeded
+     - Submits a new job run.
+     - Returns right away without starting another one.

Review Comment:
   If someone has `[state_store] clear_on_success` turned on, the run id is 
deleted when the task succeeds, so keeping task state on clear won't stop a 
resubmit. Can we add a line about that setting after the tables, like the old 
text had?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to