cjac opened a new issue #22064: URL: https://github.com/apache/airflow/issues/22064
### Apache Airflow version 2.2.3 ### What happened The queuing process in Airflow2 continues to access already-completed job, effectively corrupting start times/end times. This ruins the ability for job dependencies to chain together. After a long period of no change to completed tasks, while under heavy load, airflow re-schedules all historical jobs just to have them do no operation and complete. When the job is re-queued, the start time is re-set, and nothing happens, and the end time is immediately re-set as well. The updating of these dates to NOW() is problematic, since there are listeners monitoring the values of these fields. When the completion time is updated, other tasks are fired off. Why did the jobs get re-queued? We are able to reproduce the problem and provide a video call where questions may be asked. ### What you expected to happen Once tasks are complete, they should never be re-queued and re-executed. If tasks must be re-queued and re-executed, the start and end times of the already completed tasks must not be updated. ### How to reproduce Put environment under heavy load for a short period monitor the start/end times of tasks note that the start/end times are updated all concurrently ### Operating System composer-2.0.3-airflow-2.2.3 ### Versions of Apache Airflow Providers composer-2.0.3-airflow-2.2.3 ### Deployment Composer ### Deployment details Available on request from customer. Please be discrete with these sensitive information. ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
