Lee-W commented on code in PR #41412:
URL: https://github.com/apache/airflow/pull/41412#discussion_r1730204759
##########
airflow/providers/databricks/operators/databricks.py:
##########
@@ -123,6 +124,30 @@ def _handle_databricks_operator_execution(operator, hook,
log, context) -> None:
log.info("View run status, Spark UI, and logs at %s", run_page_url)
+def update_job_for_repair(operator, hook, job_id, run_state) -> None:
+ """
+ Update job settings(partial) to repair the run with all failed tasks.
+
+ :param operator: Databricks operator being handled
+ :param hook: Databricks hook
+ :param job_id: Job ID of Databricks
+ :param run_state: Run state of the Databricks job
+ """
+ repair_reason = next(
+ (
+ reason
+ for reason in operator.databricks_repair_reason_new_settings
+ if reason in run_state.state_message
+ ),
+ None,
+ )
+ if repair_reason is not None and
operator.databricks_repair_reason_new_settings:
Review Comment:
Would like to check whether we need the later
`operator.databricks_repair_reason_new_settings` check. When will
`repair_reason` not be none and
`operator.databricks_repair_reason_new_settings` be empty at the same time?
##########
airflow/providers/databricks/operators/databricks.py:
##########
@@ -123,6 +124,30 @@ def _handle_databricks_operator_execution(operator, hook,
log, context) -> None:
log.info("View run status, Spark UI, and logs at %s", run_page_url)
+def update_job_for_repair(operator, hook, job_id, run_state) -> None:
Review Comment:
would be great if we can improve type annotaiton
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
