Hello everyone, *TL;DR; Our CI should become a bit more stable, cheaper and snappier to run. * *But committers need to be a bit more careful when rebasing PRs from the GitHub UI.*
I just merged a change https://github.com/apache/airflow/pull/31451 that should improve the stability of PRs - especially for non-committers. It should also make much more optimized usage of our self-hosted runners. The change utilizes the fact that there are many more Public Runners available now for the Apache Software Foundation (900 parallel jobs to share between all projects) which should be enough for quite a while for all the projects of the foundation (I am monitoring it). Initially we introduced our self-hosted runners to battle queuing problems we experienced when there were 150 runners or so and we aggressively used self-hosted runners for commiter PRs and for building images for non-committers, But with 900 runners we can relax our approach and use the public runners for some of our jobs that are small (build info + waiting for images). There is a subtle change, however, when it comes to rebasing non-committer pull requests via GitHub UI by committers. Such PRs will run on self-hosted runners of ours. After the change, we are exclusively using "*GitHub Actor*" to determine which runners to run (if committer -> most of the jobs will run on self-hosted runners, if not -> everything will run using Public Runners). Till now, we used "event.pull_request.user.login" to determine it - which was basically the person who created the PR. As of now, if a committer "rebases" someone's PR, the "actor" for the rebased PR will be the commiter, not the PR owner. This means two things: * PRs rebased by committers will run on self-hosted runners * They will run much faster on much more powerful machines This is a good thing in general. Usually when a committer rebases a PR, they have a good reason (they want it to complete quickly). But also it means that when committers rebase, they need to look at the PR to make sure there is no "malicious" code that could be run on our self-hosted runners (and mine bitcoins for example). So we have to be careful what we are rebasing. J.
