potiuk commented on issue #11114:
URL: https://github.com/apache/airflow/issues/11114#issuecomment-698180016
We also recently noticed some slow-downs and limited number of slots for
Airflow. So far it's been rather generous :).
I am not exactly what the root cause was, but. It could be because of a
weird race condition with cancelling jobs. This is a special step in our build
that waits until images are build by another ("workflow_run") workflow. This is
because the PR workflow does not have the permissions to push such image, so
workflow run builds the image, but this particular job waits for it.
Usually it takes < 4 minutes, sometimes ~ 10 minutes and in rare
circumstances ~20 minutes. We do have canceling implemented in the
"workflow_run" - if it fails or gets canceled, it should cancel this "PR" job.
But there is one case wich might happen - if the "cancel" step in the
"workflow_run" gets cancelled as well, the PR might not get cancelled.
This can be easy to fix - ao far we did not have any timeout for that step,
so default 6hrs one applies. I will decrease the timeout to 40 mins or so (to
get some spare time), so that in this rare cases we will timeout faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
