AlexanderBLR opened a new pull request, #71380: URL: https://github.com/apache/airflow/pull/71380
`sync_running_jobs()` handles all active jobs in a single pass, and `sync()` wraps the whole heartbeat in one catch-all. When handling of a single job raises, the exception aborts the entire cycle: the remaining jobs are not synced and, more importantly, `attempt_submit_jobs()` never runs. If the error is deterministic for that job, the executor stops submitting tasks entirely until the scheduler is restarted. This is not hypothetical. `BatchJobCollection.pop_by_id()` raises `KeyError` before cleaning anything when the collection's bookkeeping is partially cleaned (e.g. a duplicate submission for the same workload key overwrites `key_to_id`, orphaning the older job id), so the failing job stays in `active_workers` forever and every subsequent heartbeat aborts at the same job. We hit exactly this in production (Airflow 3.2.1, `apache-airflow-providers-amazon==9.31.0`): one poisoned workload halted all Batch submissions cluster-wide (~20 Dags piling up in queued) until a scheduler restart, the only signal being `Failed to sync AwsBatchExecutor` repeating every heartbeat. The fix: - each job in `sync_running_jobs()` is handled in its own `try/except`; an unexpected per-job error evicts that job and fails its workload instead of aborting the cycle; - new `BatchJobCollection.remove_job()` used for the eviction — a defensive removal that tolerates partially cleaned state, unlike `pop_by_id()`. Credential-type boto errors (`ClientError` / `NoCredentialsError`) still reach `sync()`'s handler unchanged: they can only be raised by `_describe_jobs()` / `attempt_submit_jobs()`, which sit outside the per-job block — nothing inside the block calls AWS. Tests: two `BatchJobCollection.remove_job()` tests, eviction-with-isolation test (poisoned job evicted + failed, healthy job in the same cycle still succeeds), eviction survival when `fail()` itself raises, and a regression test for the production impact (`attempt_submit_jobs()` still runs when a tracked job cannot be synced). --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Fable 5) Generated-by: Claude Code (Fable 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
