https://ibb.co/JmYZNpk
On Fri, Nov 11, 2022 at 11:36 PM Solomon Himelbloom <[email protected]> wrote: > Attached is the graph of queue size for Apache projects where you can >> see how dramatic improvements it was (other projects of Apache do not >> have self-hosted runners as we do so they are far bigger users). > > > Do you have a link to this attachment? Thanks! > > Best, > Solomon > > > On Fri, Nov 11, 2022 at 1:17 AM Jarek Potiuk <[email protected]> wrote: > >> Hello Everyone, >> >> I thought it's a good time to share some news after yesterday's ASF >> "Builds meeting". The latest news for our CI is that ~ 2 weeks ago >> GitHub bumped the parallelism of jobs for Public runners for Apache >> Software Foundation from 300 to 900. >> >> Attached is the graph of queue size for Apache projects where you can >> see how dramatic improvements it was (other projects of Apache do not >> have self-hosted runners as we do so they are far bigger users). >> Thanks to Tobiasz Kedzierski - one of the contributors to Airflow and >> my friend who developed and maintains the graphs so that ASF can see >> some stats. >> >> This means that builds from forks of our contributors should >> experience far, far less queuing. Some new things are coming as well - >> we (ASF with our support) are talking to GitHub to get self-hosted >> runners on Azure, as well as enabling bigger instances / ARM instances >> for running our CI jobs. >> >> This will likely allow us to optimize some of the build times as well. >> I already have some ideas how we can make our builds leaner, faster >> and with less number of failures. The goal is always to get the CI >> feedback as fast as possible with as little as possible >> false-negatives (and without burning too much money that our sponsors >> give us). It's a moving target but we have some good possibilities now >> on how to make those better. >> >> Stay tuned. >> >> Also I think this is the right moment to thank both Astronomer and >> Amazon for the AWS credits (Amazon) and money (Astronomer). Again this >> year we got USD 15.000 from Amazon (last year we got USD 10.000 and >> Astronomer pays whatever we exceed above that and invested quite some >> time (mostly of Ash) to get our "VM" infrastructure up and running. >> >> There are also likely more news coming in the light of AIP-47 - where >> we are working with Amazon, Google, Databricks, Snowflake (and others) >> to make our "System" test infrastructure even stronger and be able to >> automatically test the stability and regressions of integrations of >> Airflow. >> >> J. >> >
