AlexanderBLR commented on code in PR #71380:
URL: https://github.com/apache/airflow/pull/71380#discussion_r3763894365
##########
providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py:
##########
@@ -268,11 +268,30 @@ def sync_running_jobs(self):
self.log.debug("Active Workers: %s", describe_job_response)
for job in describe_job_response:
- if job.get_job_state() == State.FAILED:
- self._handle_failed_job(job)
- elif job.get_job_state() == State.SUCCESS:
- workload_key = self.active_workers.pop_by_id(job.job_id)
- self.success(workload_key)
+ try:
+ if job.get_job_state() == State.FAILED:
+ self._handle_failed_job(job)
+ elif job.get_job_state() == State.SUCCESS:
+ workload_key = self.active_workers.pop_by_id(job.job_id)
+ self.success(workload_key)
+ except Exception:
+ self.log.exception(
+ "Evicting Batch job %s after an unexpected error while
syncing it, "
+ "so that one broken job cannot abort the sync cycle and
stall task "
+ "submission for the whole executor.",
+ job.job_id,
+ )
+ self._evict_job(job.job_id)
+
+ def _evict_job(self, job_id: str) -> None:
+ """Remove a job from the collection and fail its workload, tolerating
corrupted bookkeeping."""
+ workload_key = self.active_workers.remove_job(job_id)
+ if workload_key is None:
+ return
+ try:
+ self.fail(workload_key)
Review Comment:
It's possible, yes — the narrow case is _"the job actually succeeded in
Batch, but `success()` itself raised"._ By the time we're in the eviction path,
reporting success has already failed, so the realistic options are: leave the
task instance with no terminal state (it lingers until the scheduler's
stuck-task handling times it out, and typically ends up failed anyway, just
slower), or fail it promptly so the normal retry policy takes over. `fail()`
seemed the least-bad default — deterministic, faster recovery, and within
Airflow's usual contract that retries may re-run a task. But if you see a
better terminal state to report here, happy to change it.
---Drafted-by: Claude Code (Fable 5); reviewed by @AlexanderBLR before
posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
