amoghrajesh commented on code in PR #76295: URL: https://github.com/apache/airflow/pull/76295#discussion_r4453336940
########## providers/databricks/docs/operators/notebook.rst: ########## @@ -80,3 +81,35 @@ attempt is observed, so downstream failure handling is not delayed by long-runni Only unlimited retries (``max_retries=-1``) keep the Airflow task waiting (or deferring) until the parent workflow run reaches a terminal state, because Databricks may still launch a retry attempt under the same ``task_key`` until then. Sibling tasks in the run continue independently. + + +Durable execution +----------------- + +Outside a +:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`, +this operator submits a run and polls it to completion on the worker. By default it runs in a +*durable* mode: the Databricks run id is persisted to +:doc:`task state store <apache-airflow:core-concepts/task-state-store>` before polling begins, so if +the worker crashes or is preempted and the task is retried, the operator reconnects to the run that +is already executing on Databricks instead of submitting a duplicate. + +On retry the operator checks the prior run's state: + +* if it is still running, the operator reconnects and continues polling +* if it already succeeded, the operator returns immediately without resubmitting +* if it failed terminally, the operator submits a fresh run + +Pass ``durable=False`` to always submit a fresh run on retry. + +Durable execution requires Airflow 3.3 or newer, since it relies on the task state store. On +earlier Airflow versions the flag is a no-op and the operator always submits a fresh run on retry. +If the task state store is unavailable at runtime, the operator logs that crash recovery is +disabled and behaves the same way. + +Inside a ``DatabricksWorkflowTaskGroup`` nothing changes: the run is submitted by the workflow +launch task and this operator only monitors its own task within that shared run, so a retry already +reconnects rather than resubmitting. + +See :doc:`apache-airflow:core-concepts/resumable-tasks` for how clearing a task interacts with the +stored run id, and for the retention caveats around ``airflow state-store clean``. Review Comment: Inlined the retention paragraph the way `submit_run.rst` does it and pointed it at `administration-and-deployment/task-and-asset-state-store-cleanup`. The clearing half still points at `resumable-tasks` and now mentions `[state_store] clear_on_success`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
