potiuk commented on pull request #17514: URL: https://github.com/apache/airflow/pull/17514#issuecomment-895468312
Let's see if it will solve the problem of "sometimes-hanging" jobs at test stage. If it does I REALLY think we should always spin out a new fresh runner for every job @ashb - I think (and always thought) that re-using running machines between the jobs is a bad idea, there are always left-overs and especially when you cancel a running job, the runners might end up in some weird state which might cause all subsequent jobs for this runner to fail. There are many reasons why some left-overs might spoil subsequent runs, and it's next to impossible to foresee and cleanup all of them (and also it's never future-proof - people will continue on adding stuff that will not get cleaned eventually and it's a continuous struggle. I really think all our jobs should ALWAYS start runner from totally clean state - even if it means that we have to start a new VM for a job that lasts few seconds and it adds significant overhead. That (From my experience) saves hundreds of hours of failed jobs, retries, re-runs and people trying to investigate and fix the issue. Reusing anything for CI job from previous run is always a bad idea and we should always start from scratch IMHO. Especially the amount of frustration it creates for contributors is huge and people maintaining it cannot be blamed for it (as this is a constant struggle). I will finally maybe have a look at that for GCP and I will try to make it works in the way that we always have only one pass for each runner and we kill the vm right after, without trying to keep the pool of those and trying to process more jobs in a loop. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
