Awesome, thanks Rafael. On Tue, Apr 13, 2021, 20:33 Jarek Potiuk <[email protected]> wrote:
> Fantastic news Rafał! Thank you and the Composer team for that ! > > On Tue, Apr 13, 2021 at 9:27 PM Rafal Biegacz > <[email protected]> wrote: > >> Hi, >> >> During the weekend I created a new GCP project for Apache Airflow CI/CD >> purposes and I provided access to this project to Jarek et al. >> >> Right now this project is linked to a billing account that has $8000 of >> GCP credits. It should be good enough for us to start replicating the >> CI/CD/testing infrastructure on GCP. >> >> Regards, Rafal. >> Cloud Composer <https://cloud.google.com/composer> >> >> >> >> >> On Sat, Apr 10, 2021 at 2:51 PM Jarek Potiuk <[email protected]> wrote: >> >>> Hello everyone, >>> >>> Probably some of you noticed (especially committers but also regular >>> contributors) that our CI got quite a bit speedier over the last couple of >>> months and also quite a bit more stable. >>> >>> But we need a bit more cloud credits to keep it this way :(. I am >>> reaching out to the community with the kind usk, because maybe there are >>> some sources of credits that we can tap into. It would be great if the >>> community of Airflow could help with that :D. >>> >>> The speed-up has been a joined effort of a number of people - mostly Ash >>> who built the whole auto-scaling infrastructure for it and customized the >>> Github Runner to safely run it on our self-hosted infrastructure, and >>> myself working on optimizing and parallelizing the CI builds (which also >>> got much nicer and more usable, colorful output). >>> >>> However that was at the cost of ... credits from AWS. Unfortunately the >>> burn rate turned quite a bit more than we (my estimations actually) >>> initially anticipated and we are running out of those credits pretty >>> quickly. At the current rate (which I still hope to bring down soon) we >>> will run out the current credits by mid May (or earlier, depending on how >>> many PRs we'll get merged :) ). >>> >>> We are working on getting more credits (keep fingers crossed) from both >>> AWS and GCP teams (there we will need to replicate the infrastructure we >>> have in AWS now). >>> >>> I also have a very fresh idea from the Apache Spark team >>> https://github.com/apache/spark/pull/32092 that resulted from our >>> discussions at [email protected]. We are going to explore which might >>> help to drive the cost down significantly (more on that soon). So >>> eventually the regular cost of the infrastructure might be much cheaper, >>> but this might take a bit of time to get deployed. >>> >>> If you have some ideas, sources of credits (AWS but GCP as well) please >>> reach out to me personally. We can make a very good use of those. >>> >>> J. >>> >>> >>> -- >>> +48 660 796 129 <+48%20660%20796%20129> >>> >> > > -- > +48 660 796 129 >
