kaxil commented on code in PR #76295:
URL: https://github.com/apache/airflow/pull/76295#discussion_r4515425157


##########
providers/databricks/docs/operators/notebook.rst:
##########
@@ -80,3 +82,57 @@ attempt is observed, so downstream failure handling is not 
delayed by long-runni
 Only unlimited retries (``max_retries=-1``) keep the Airflow task waiting (or 
deferring) until the
 parent workflow run reaches a terminal state, because Databricks may still 
launch a retry attempt
 under the same ``task_key`` until then. Sibling tasks in the run continue 
independently.
+
+
+Durable execution
+-----------------
+
+Outside a
+:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`,
+this operator submits a run and then waits for it, on the worker or in the 
Triggerer when
+``deferrable=True``. By default it runs in a *durable* mode: the Databricks 
run id is persisted to
+:doc:`task state store <apache-airflow:core-concepts/task-state-store>` before 
the wait begins, so if
+the worker crashes or is preempted and the task is retried, the operator 
reconnects to the run that
+is already executing on Databricks instead of submitting a duplicate.
+
+On retry the operator checks the prior run's state:
+
+* if it is still running, the operator reconnects and continues polling
+* if it already succeeded, the operator returns immediately without 
resubmitting
+* if it failed terminally, the operator submits a fresh run
+
+Pass ``durable=False`` to always submit a fresh run on retry.
+
+Unlike 
:class:`~airflow.providers.databricks.operators.databricks.DatabricksSubmitRunOperator`
 and
+:class:`~airflow.providers.databricks.operators.databricks.DatabricksRunNowOperator`,
 where
+``durable`` has no effect in deferrable mode, durable execution here also 
covers
+``deferrable=True``: the run id is stored before the task defers, so a try 
lost while the Triggerer
+was waiting still reconnects on the next one.
+
+Durable execution requires Airflow 3.3 or newer, since it relies on the task 
state store. On
+earlier Airflow versions the flag is a no-op and the operator always submits a 
fresh run on retry.
+If the task state store is unavailable at runtime, the operator logs that 
crash recovery is
+disabled and behaves the same way.
+
+Inside a ``DatabricksWorkflowTaskGroup`` nothing changes: the run is submitted 
by the workflow
+launch task and this operator only monitors its own task within that shared 
run, so a retry already
+reconnects rather than resubmitting.
+
+The stored run id is not deleted automatically; that only happens when someone 
runs
+``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs 
in between, the run
+id is gone by the next retry and the operator submits a fresh run instead of 
reconnecting. Avoid
+running cleanup on a schedule shorter than your longest ``retry_delay``. See
+:doc:`apache-airflow:administration-and-deployment/task-and-asset-state-store-cleanup`.
+
+Which clear you use decides whether the stored run id survives. Clearing a 
single task from the

Review Comment:
   `clearTaskInstances` discards task state for every task instance it clears 
unless `keep_task_state` is set, so a task group clear, a clear with upstream 
or downstream ticked, and `airflowctl dags clear` all drop the id too, not only 
a single-task clear. `core-concepts/resumable-tasks` already has the exact 
list. Could this link there instead of restating it? Same in `task.rst`.



##########
providers/databricks/docs/operators/notebook.rst:
##########
@@ -80,3 +82,57 @@ attempt is observed, so downstream failure handling is not 
delayed by long-runni
 Only unlimited retries (``max_retries=-1``) keep the Airflow task waiting (or 
deferring) until the
 parent workflow run reaches a terminal state, because Databricks may still 
launch a retry attempt
 under the same ``task_key`` until then. Sibling tasks in the run continue 
independently.
+
+
+Durable execution
+-----------------
+
+Outside a
+:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`,
+this operator submits a run and then waits for it, on the worker or in the 
Triggerer when
+``deferrable=True``. By default it runs in a *durable* mode: the Databricks 
run id is persisted to
+:doc:`task state store <apache-airflow:core-concepts/task-state-store>` before 
the wait begins, so if
+the worker crashes or is preempted and the task is retried, the operator 
reconnects to the run that
+is already executing on Databricks instead of submitting a duplicate.
+
+On retry the operator checks the prior run's state:
+
+* if it is still running, the operator reconnects and continues polling
+* if it already succeeded, the operator returns immediately without 
resubmitting
+* if it failed terminally, the operator submits a fresh run
+
+Pass ``durable=False`` to always submit a fresh run on retry.
+
+Unlike 
:class:`~airflow.providers.databricks.operators.databricks.DatabricksSubmitRunOperator`
 and
+:class:`~airflow.providers.databricks.operators.databricks.DatabricksRunNowOperator`,
 where
+``durable`` has no effect in deferrable mode, durable execution here also 
covers
+``deferrable=True``: the run id is stored before the task defers, so a try 
lost while the Triggerer
+was waiting still reconnects on the next one.
+
+Durable execution requires Airflow 3.3 or newer, since it relies on the task 
state store. On
+earlier Airflow versions the flag is a no-op and the operator always submits a 
fresh run on retry.
+If the task state store is unavailable at runtime, the operator logs that 
crash recovery is
+disabled and behaves the same way.
+
+Inside a ``DatabricksWorkflowTaskGroup`` nothing changes: the run is submitted 
by the workflow
+launch task and this operator only monitors its own task within that shared 
run, so a retry already
+reconnects rather than resubmitting.
+
+The stored run id is not deleted automatically; that only happens when someone 
runs
+``airflow state-store clean``. If a task's ``retry_delay`` is longer than
+``[state_store] default_retention_days`` (30 days by default) and cleanup runs 
in between, the run
+id is gone by the next retry and the operator submits a fresh run instead of 
reconnecting. Avoid

Review Comment:
   Is the cleanup schedule the right rule here? `expires_at` is set when the id 
is written, so a retry that comes more than `default_retention_days` after the 
submit can lose the id with any cleanup schedule. The "not deleted 
automatically" line above is also off now that the clears below and 
`clear_on_success` delete it. Same in `task.rst`, and in `submit_run.rst` where 
this paragraph came from.



##########
providers/databricks/src/airflow/providers/databricks/operators/databricks.py:
##########
@@ -82,6 +82,62 @@
 XCOM_RUN_PAGE_URL_KEY = "run_page_url"
 XCOM_STATEMENT_ID_KEY = "statement_id"
 
+# Databricks splits run state across life_cycle_state and result_state, while 
the ResumableJobMixin
+# status interface is a single string. The durable operators encode both as 
"LIFE_CYCLE:RESULT" and
+# decode them with the predicates below, so all three share one view of which 
runs can be rejoined.
+RUN_NOT_FOUND_STATUS = "NOT_FOUND:"
+TERMINATING_SETTLE_TIMEOUT_SECONDS = 60
+ACTIVE_RUN_LIFE_CYCLE_STATES = (
+    "PENDING",
+    "QUEUED",
+    "RUNNING",
+    "TERMINATING",
+    "BLOCKED",
+    "WAITING_FOR_RETRY",
+)
+
+
+def fetch_run_status(
+    hook: DatabricksHook,
+    run_id: JsonValue,
+    *,
+    poll_interval: float = 5,
+    settle_timeout: float = TERMINATING_SETTLE_TIMEOUT_SECONDS,
+) -> str:
+    """
+    Return the encoded status of ``run_id``, or a sentinel if its run history 
is gone.
+
+    ``TERMINATING`` is waited out because it does not say whether the run is 
finishing on its own or
+    being cancelled, and ``on_kill`` cancels asynchronously without clearing 
the stored id. On
+    timeout it is reported as-is, which counts as active: rejoining a slow run 
beats duplicating it.
+    """
+    deadline = time.monotonic() + settle_timeout
+    while True:
+        try:
+            # The stored external id is always a Databricks run id; the mixin 
types it as JsonValue.
+            run_info = hook.get_run(cast("int", run_id))
+        except DatabricksApiError as e:
+            # A stored run whose history expired (or was deleted) returns 404. 
The sentinel is neither
+            # active nor succeeded, so the mixin resubmits fresh instead of 
failing the task.
+            if e.http_status_code == 404:
+                return RUN_NOT_FOUND_STATUS
+            raise
+        state = RunState(**run_info["state"])
+        if state.life_cycle_state != "TERMINATING" or time.monotonic() >= 
deadline:
+            return f"{state.life_cycle_state}:{state.result_state}"
+        time.sleep(poll_interval)

Review Comment:
   The deadline is only checked after `get_run`, so this sleeps a full 
`poll_interval` past it. With `polling_period_seconds=300` on SubmitRun or 
RunNow a stored run in `TERMINATING` holds the worker for 300s before the retry 
decides, and with 0 it calls `runs/get` back to back for the whole minute. 
Could this sleep `min(poll_interval, remaining)` with a small floor, and the 
stuck-run test use a `poll_interval` larger than `settle_timeout`?



##########
providers/databricks/tests/unit/databricks/operators/test_databricks.py:
##########
@@ -5267,3 +5269,293 @@ def 
test_monitor_waits_out_waiting_for_retry_state(self, mock_databricks_hook, m
 
         operator.monitor_databricks_job()
         assert mock_databricks_hook.return_value.get_run.call_count == 3
+
+
[email protected](
+    not AIRFLOW_V_3_3_PLUS, reason="task_state_store (durable execution) 
requires Airflow 3.3+"
+)
+class TestDatabricksTaskBaseOperatorDurable:
+    @pytest.fixture(params=["notebook", "task"])
+    def make_operator(self, request):
+        def _make(**kwargs):
+            if request.param == "notebook":
+                return DatabricksNotebookOperator(
+                    task_id=TASK_ID,
+                    notebook_path="/test",
+                    source="WORKSPACE",
+                    new_cluster=NEW_CLUSTER,
+                    **kwargs,
+                )
+            return DatabricksTaskOperator(
+                task_id=TASK_ID,
+                task_config={"notebook_task": NOTEBOOK_TASK},
+                new_cluster=NEW_CLUSTER,
+                **kwargs,
+            )
+
+        return _make
+
+    @staticmethod
+    def _task_store(stored_run_id=None):
+        task_store = MagicMock(spec_set=["get", "set"])
+        task_store.get.return_value = stored_run_id
+        return task_store
+
+    
@mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook")
+    def test_persists_run_id_on_fresh_submit(self, db_mock_class, 
make_operator):

Review Comment:
   Nothing checks that the run id is stored before the page URL lookup. Adding 
a `get_run_page_url` call back into `_launch_job` before it returns, which 
strands the run if the lookup fails, keeps all 303 tests in this file green. 
Could one test make `get_run_page_url` raise and assert `task_state_store.set` 
was still called with the run id?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to