vincbeck commented on code in PR #77158:
URL: https://github.com/apache/airflow/pull/77158#discussion_r4601795119


##########
providers/amazon/docs/operators/batch.rst:
##########
@@ -51,6 +51,22 @@ use 
:class:`~airflow.providers.amazon.aws.operators.batch.BatchOperator`.
     :start-after: [START howto_operator_batch]
     :end-before: [END howto_operator_batch]
 
+Durable execution
+=================
+
+By default ``BatchOperator`` saves the Batch job id to the
+:doc:`task state store <apache-airflow:core-concepts/task-state-store>` right 
after submitting the
+job. If the worker crashes and the task is retried, the operator checks that 
job first:

Review Comment:
   `on_kill` still calls `terminate_job` on SIGTERM and on `execution_timeout`. 
So after a pod eviction or a timeout, the retry finds a killed job and 
resubmits, or with a short `retry_delay` reconnects to a job that is still 
shutting down and fails. Should we mention that durable only helps when the 
worker dies without SIGTERM (OOM kill, node loss)? Or add a flag like Glue's 
`stop_job_run_on_kill`?



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -363,6 +368,40 @@ def submit_job(self, context: Context):
             aws_partition=self.hook.conn_partition,
             job_id=self.job_id,
         )
+        return self.job_id
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        jobs = self.hook.client.describe_jobs(jobs=[cast("str", 
external_id)])["jobs"]
+        return jobs[0]["status"] if jobs else "NOT_FOUND"
+
+    def is_job_active(self, status: str) -> bool:
+        return status in self.hook.INTERMEDIATE_STATES
+
+    def is_job_succeeded(self, status: str) -> bool:
+        return status == self.hook.SUCCESS_STATE
+
+    def poll_until_complete(self, external_id: JsonValue, context: Context) -> 
None:
+        self._attach_to_job(external_id, context)
+        if self.wait_for_completion:
+            self.monitor_job(context)
+
+    def get_job_result(self, external_id: JsonValue, context: Context) -> str 
| None:
+        self._attach_to_job(external_id, context)
+        return self.job_id
+
+    def _attach_to_job(self, external_id: JsonValue, context: Context) -> None:
+        # None comes from subclasses whose submit_job override predates it 
returning the job id;
+        # they still set self.job_id, so keep it.

Review Comment:
   With such a subclass the job id is never saved to the task state store (the 
mixin skips `set` when `submit_job` returns `None`), so durability is silently 
off. Should we at least log a warning here?



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -363,6 +368,40 @@ def submit_job(self, context: Context):
             aws_partition=self.hook.conn_partition,
             job_id=self.job_id,
         )
+        return self.job_id
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        jobs = self.hook.client.describe_jobs(jobs=[cast("str", 
external_id)])["jobs"]
+        return jobs[0]["status"] if jobs else "NOT_FOUND"
+
+    def is_job_active(self, status: str) -> bool:
+        return status in self.hook.INTERMEDIATE_STATES
+
+    def is_job_succeeded(self, status: str) -> bool:
+        return status == self.hook.SUCCESS_STATE
+
+    def poll_until_complete(self, external_id: JsonValue, context: Context) -> 
None:
+        self._attach_to_job(external_id, context)
+        if self.wait_for_completion:
+            self.monitor_job(context)
+
+    def get_job_result(self, external_id: JsonValue, context: Context) -> str 
| None:
+        self._attach_to_job(external_id, context)
+        return self.job_id

Review Comment:
   nit: on the already-succeeded path `monitor_job` is skipped, so this try 
ends without the Job definition, Job queue and CloudWatch links (XComs are 
cleared at each try). Maybe call `_persist_links` and 
`_persist_cloudwatch_link` here when the job was not monitored?



##########
providers/amazon/src/airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -363,6 +368,40 @@ def submit_job(self, context: Context):
             aws_partition=self.hook.conn_partition,
             job_id=self.job_id,
         )
+        return self.job_id
+
+    def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+        jobs = self.hook.client.describe_jobs(jobs=[cast("str", 
external_id)])["jobs"]
+        return jobs[0]["status"] if jobs else "NOT_FOUND"
+
+    def is_job_active(self, status: str) -> bool:
+        return status in self.hook.INTERMEDIATE_STATES
+
+    def is_job_succeeded(self, status: str) -> bool:
+        return status == self.hook.SUCCESS_STATE
+
+    def poll_until_complete(self, external_id: JsonValue, context: Context) -> 
None:
+        self._attach_to_job(external_id, context)
+        if self.wait_for_completion:
+            self.monitor_job(context)
+
+    def get_job_result(self, external_id: JsonValue, context: Context) -> str 
| None:
+        self._attach_to_job(external_id, context)
+        return self.job_id
+
+    def _attach_to_job(self, external_id: JsonValue, context: Context) -> None:
+        # None comes from subclasses whose submit_job override predates it 
returning the job id;
+        # they still set self.job_id, so keep it.
+        if external_id is None or self.job_id == external_id:
+            return
+        self.job_id = cast("str", external_id)
+        BatchJobDetailsLink.persist(

Review Comment:
   nit: this `BatchJobDetailsLink.persist` block is the same as the one in 
`submit_job`, any chance to factorize?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to