jscheffl opened a new pull request, #66995: URL: https://github.com/apache/airflow/pull/66995
In our farm we have seen a race in push_logs function that after `aio_logfile.exists()` the call to `aio_logfile.stat()` failed because file vanished in between. What had been a defensive programming turned out to be a concurrency race in very low probability. Adding more resiliency to prevent exception being raised when log file dis-appears while check and open() call if the logging back-end makes uploads and deletes local copy. Additionally alongside realized that any kind of exception then will raise an exception in `fetch_and_run_job()` which potentially causes to skip the call to `self.jobs.remove(job)` which then makes the worker believe the job is still active but it actually had terminated. Then worker will never cleanly shut down (except if a drain timeout is set) and also might not pick new jobs if capacity is calculated wrongly. FYI @AutomationDev85 @wolfdn @renat-sagut --- ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by". --> - [ ] Yes (please specify the tool below) <!-- Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --> --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
