One important comment: I think we need to build a small team of volunteers to take care of our "AWS" account installations. We want to avoid SPOF and for that - experience with breeze/dev env is that we need 4-5 people somewhat monitoring what's going on and reacting - to account for vacations and the like. Since they will also need access to the AWS account, this brings certain responsibilities.
We will likely have soon few more proposals related - so it would be great to have a dedicated channel on slack with notifications (like we have #internal-airflow-ci-cd) - and people subscribing there, receiving notifications. I will propose a small AIP to add automated and self-learning monitoring for the account. And with the AI agents doind bulk of the work, I **think** overhead to learn and respond to any issues should be minimal. J . On Fri, Sep 18, 2026 at 11:20 AM Jarek Potiuk <[email protected]> wrote: > Really cool. Great to see it taking shape :). I made a few comments - but > it looks **great** - we just need RunsOn evaluation - I think it's an > interesting (0-maintenance) alternative to our own EKS cluster. > > On Fri, Sep 18, 2026 at 10:52 AM Shahar Epstein <[email protected]> wrote: > >> Hello everyone, >> >> Activity in Apache Airflow has skyrocketed over the past year (thanks to >> both humans and their carbon-wasting assistants) and the load on our >> GitHub-hosted CI runners has increased accordingly. >> More often than not, we experience slowness and hiccups with the >> GitHub-hosted runners, primarily because of the number of concurrent jobs. >> This is especially frustrating for maintainers when scheduled canary runs >> need to be restarted or when a PR addressing a high-priority issue, such >> as >> a security issue, needs to be validated quickly. >> Furthermore, because we share the GitHub-hosted runner capacity with the >> entire ASF organization, Airflow’s usage may also affect other Apache >> projects. >> >> While we continue exploring ways to reduce our CI footprint through code >> and workflow optimizations, we have also investigated offloading some jobs >> to self-hosted runners in cloud environments. This idea has been discussed >> on the dev list several times over the years ([1] >> <https://lists.apache.org/thread/8htrdgf2h8qz1hv7mbb96v8l8x8d1dyl>, [2] >> <https://lists.apache.org/thread/lsnpdovfpnj81pwdlhk768bv8off4nd1>, [3] >> <https://lists.apache.org/thread/55z686pt7377wt21yqjqj26y9s81zcjo>, [4] >> <https://lists.apache.org/thread/ogwjy38hyly9tksfzl294bbp434jokf3>, [5] >> <https://lists.apache.org/thread/4okht98xl127mc9nynpyzxrjdrvj8h0c>), and >> several proofs of concept have been developed. However, it has not yet >> materialized into a permanent solution, partly because of competing >> priorities and partly because optimizations made at the time were >> sufficient. >> >> The problem has continued to grow, so I have decided to tackle it again. >> >> For those who are not aware, Apache Airflow participates in the AWS Open >> Source Credits Program, through which we receive credits once in two >> years. >> These credits are currently used primarily to host our documentation on >> S3. >> We recently asked AWS for additional credits to support the self-hosted >> runner effort, and they generously agreed to contribute them. >> >> AIP-118 <https://cwiki.apache.org/confluence/x/-JXwGg> describes the >> motivation for using self-hosted CI runners, the proposed policy governing >> their use, and the available implementation alternatives. I have also >> included an appendix >> < >> https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974672/AIP-118+Appendix+%E2%80%94+evidence+and+implementation+details?src=contextnavpagetreemode >> > >> containing supporting data and implementation details. >> >> In general, self-hosted runners would be limited to trusted runs: >> scheduled >> workflows and runs explicitly approved by committers. >> >> Because of the current budget constraints, we would initially offload >> scheduled runs and selected runs manually labeled by committers. If this >> proves successful, we may ask AWS for additional credits in the future, >> allowing all committer-triggered runs to use self-hosted runners by >> default. >> >> So far, I have tested two implementation approaches: >> >> *- EKS:* A Kubernetes cluster using Spot Instances. This is cost-effective >> but requires us to operate and maintain the cluster. >> >> *- CodeBuild-managed GitHub Actions runners:* This requires significantly >> less maintenance, but it is x7 times as expensive as the EKS option. I >> have >> therefore ruled it out for now. >> I am also evaluating a 3rd option: RunsOn <https://runs-on.com/>. It >> appears to provide many of the advantages of the EKS approach with a much >> simpler infrastructure setup and may cost less. However, it would >> introduce >> reliance on a third party and requires public acknowledgement of its use. >> >> AIP-118 does not propose any changes affecting Apache Airflow users. >> Nevertheless, because it would significantly affect the contribution >> workflows, I believe it is appropriate to put it to a formal vote. I would >> be happy to hear any concerns, suggestions, or alternative ideas in this >> thread before going into a vote. >> >> I'd like to thank: >> - Hussein, Jarek, and Ash for their previous experience with self-hosted >> runners, which was invaluable in formulating the AIP and developing the >> EKS >> design. >> - Niko for his tremendous help in securing the additional AWS credits. >> >> >> Shahar >> >
