Really cool. Great to see it taking shape :). I made a few comments - but it looks **great** - we just need RunsOn evaluation - I think it's an interesting (0-maintenance) alternative to our own EKS cluster.
On Fri, Sep 18, 2026 at 10:52 AM Shahar Epstein <[email protected]> wrote: > Hello everyone, > > Activity in Apache Airflow has skyrocketed over the past year (thanks to > both humans and their carbon-wasting assistants) and the load on our > GitHub-hosted CI runners has increased accordingly. > More often than not, we experience slowness and hiccups with the > GitHub-hosted runners, primarily because of the number of concurrent jobs. > This is especially frustrating for maintainers when scheduled canary runs > need to be restarted or when a PR addressing a high-priority issue, such as > a security issue, needs to be validated quickly. > Furthermore, because we share the GitHub-hosted runner capacity with the > entire ASF organization, Airflow’s usage may also affect other Apache > projects. > > While we continue exploring ways to reduce our CI footprint through code > and workflow optimizations, we have also investigated offloading some jobs > to self-hosted runners in cloud environments. This idea has been discussed > on the dev list several times over the years ([1] > <https://lists.apache.org/thread/8htrdgf2h8qz1hv7mbb96v8l8x8d1dyl>, [2] > <https://lists.apache.org/thread/lsnpdovfpnj81pwdlhk768bv8off4nd1>, [3] > <https://lists.apache.org/thread/55z686pt7377wt21yqjqj26y9s81zcjo>, [4] > <https://lists.apache.org/thread/ogwjy38hyly9tksfzl294bbp434jokf3>, [5] > <https://lists.apache.org/thread/4okht98xl127mc9nynpyzxrjdrvj8h0c>), and > several proofs of concept have been developed. However, it has not yet > materialized into a permanent solution, partly because of competing > priorities and partly because optimizations made at the time were > sufficient. > > The problem has continued to grow, so I have decided to tackle it again. > > For those who are not aware, Apache Airflow participates in the AWS Open > Source Credits Program, through which we receive credits once in two years. > These credits are currently used primarily to host our documentation on S3. > We recently asked AWS for additional credits to support the self-hosted > runner effort, and they generously agreed to contribute them. > > AIP-118 <https://cwiki.apache.org/confluence/x/-JXwGg> describes the > motivation for using self-hosted CI runners, the proposed policy governing > their use, and the available implementation alternatives. I have also > included an appendix > < > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974672/AIP-118+Appendix+%E2%80%94+evidence+and+implementation+details?src=contextnavpagetreemode > > > containing supporting data and implementation details. > > In general, self-hosted runners would be limited to trusted runs: scheduled > workflows and runs explicitly approved by committers. > > Because of the current budget constraints, we would initially offload > scheduled runs and selected runs manually labeled by committers. If this > proves successful, we may ask AWS for additional credits in the future, > allowing all committer-triggered runs to use self-hosted runners by > default. > > So far, I have tested two implementation approaches: > > *- EKS:* A Kubernetes cluster using Spot Instances. This is cost-effective > but requires us to operate and maintain the cluster. > > *- CodeBuild-managed GitHub Actions runners:* This requires significantly > less maintenance, but it is x7 times as expensive as the EKS option. I have > therefore ruled it out for now. > I am also evaluating a 3rd option: RunsOn <https://runs-on.com/>. It > appears to provide many of the advantages of the EKS approach with a much > simpler infrastructure setup and may cost less. However, it would introduce > reliance on a third party and requires public acknowledgement of its use. > > AIP-118 does not propose any changes affecting Apache Airflow users. > Nevertheless, because it would significantly affect the contribution > workflows, I believe it is appropriate to put it to a formal vote. I would > be happy to hear any concerns, suggestions, or alternative ideas in this > thread before going into a vote. > > I'd like to thank: > - Hussein, Jarek, and Ash for their previous experience with self-hosted > runners, which was invaluable in formulating the AIP and developing the EKS > design. > - Niko for his tremendous help in securing the additional AWS credits. > > > Shahar >
