amoghrajesh commented on code in PR #76295:
URL: https://github.com/apache/airflow/pull/76295#discussion_r4520640331


##########
providers/databricks/src/airflow/providers/databricks/operators/databricks.py:
##########
@@ -82,6 +82,62 @@
 XCOM_RUN_PAGE_URL_KEY = "run_page_url"
 XCOM_STATEMENT_ID_KEY = "statement_id"
 
+# Databricks splits run state across life_cycle_state and result_state, while 
the ResumableJobMixin
+# status interface is a single string. The durable operators encode both as 
"LIFE_CYCLE:RESULT" and
+# decode them with the predicates below, so all three share one view of which 
runs can be rejoined.
+RUN_NOT_FOUND_STATUS = "NOT_FOUND:"
+TERMINATING_SETTLE_TIMEOUT_SECONDS = 60
+ACTIVE_RUN_LIFE_CYCLE_STATES = (
+    "PENDING",
+    "QUEUED",
+    "RUNNING",
+    "TERMINATING",
+    "BLOCKED",
+    "WAITING_FOR_RETRY",
+)
+
+
+def fetch_run_status(
+    hook: DatabricksHook,
+    run_id: JsonValue,
+    *,
+    poll_interval: float = 5,
+    settle_timeout: float = TERMINATING_SETTLE_TIMEOUT_SECONDS,
+) -> str:
+    """
+    Return the encoded status of ``run_id``, or a sentinel if its run history 
is gone.
+
+    ``TERMINATING`` is waited out because it does not say whether the run is 
finishing on its own or
+    being cancelled, and ``on_kill`` cancels asynchronously without clearing 
the stored id. On
+    timeout it is reported as-is, which counts as active: rejoining a slow run 
beats duplicating it.
+    """
+    deadline = time.monotonic() + settle_timeout
+    while True:
+        try:
+            # The stored external id is always a Databricks run id; the mixin 
types it as JsonValue.
+            run_info = hook.get_run(cast("int", run_id))
+        except DatabricksApiError as e:
+            # A stored run whose history expired (or was deleted) returns 404. 
The sentinel is neither
+            # active nor succeeded, so the mixin resubmits fresh instead of 
failing the task.
+            if e.http_status_code == 404:
+                return RUN_NOT_FOUND_STATUS
+            raise
+        state = RunState(**run_info["state"])
+        if state.life_cycle_state != "TERMINATING" or time.monotonic() >= 
deadline:
+            return f"{state.life_cycle_state}:{state.result_state}"
+        time.sleep(poll_interval)

Review Comment:
   I like this one! Now sleeps `max(MIN_SETTLE_POLL_SECONDS, min(poll_interval, 
remaining))`, with remaining computed from the deadline each iteration.
   
   Added two tests, both with `poll_interval` above `settle_timeout`: one pins 
the clamp to the remaining time, the other pins the floor at `poll_interval=0`. 
Both go red against the old `sleep(poll_interval)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to