+1 for this. I do not yet have enough chance to experience many job failures, but it won’t harm us to test them out. Plus, it saves some of the cost.
Best, Wei > On Apr 5, 2024, at 11:36 PM, Jarek Potiuk <ja...@potiuk.com> wrote: > > Seeing no big "no's" - I will prepare and run the experiment - starting > some time next week, after we get 2.9.0 out - I do not want to break > anything there. In the meantime, preparatory PR to add "use self-hosted > runners" label is out https://github.com/apache/airflow/pull/38779 > > On Fri, Apr 5, 2024 at 4:21 PM Bishundeo, Rajeshwar > <rbish...@amazon.com.invalid> wrote: > >> +1 with trying this out. I agree with keeping the canary builds >> self-hosted in order to validate the usage for the PRs. >> >> -- Rajesh >> >> >> From: Jarek Potiuk <ja...@potiuk.com> >> Reply-To: "dev@airflow.apache.org" <dev@airflow.apache.org> >> Date: Friday, April 5, 2024 at 8:36 AM >> To: "dev@airflow.apache.org" <dev@airflow.apache.org> >> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Consider disabling >> self-hosted runners for commiter PRs >> >> >> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >> >> >> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que >> le contenu ne présente aucun risque. >> >> >> Yeah. Valid concerns Hussein. >> >> And I am happy to share some more information on that. I did not want to >> put all of that in the original email, but I see that might be interesting >> for you and possibly others. >> >> I am closely following the numbers now. One of the reasons I am doing / >> proposing it now is that finally (after almost 3 years of waiting) we >> finally have access to some metrics that we can check. As of last week I >> got access to the ASF metrics ( >> https://issues.apache.org/jira/browse/INFRA-25662). >> >> I have access to "organisation" level information. Infra does not want to >> open it to everyone - even to every member - but since I got very active >> and been helping with a number I got the access granted as an exception. >> Also I saw a small dashboard the INFRA prepares to open to everyone once >> they sort the access where we will be able to see the "per-project" usage. >> >> Some stats that I can share (they asked not to share too much). >> >> From what I looked at I can tell that we are right now (the whole ASF >> organisation) safely below the total capacity. With a large margin - enough >> to handle spikes, but of course the growth of usage is there and if >> uncontrolled - we can again reach the same situation that triggered getting >> self-hosted runners a few years ago. >> >> Luckily - INRA gets it under control this time |(and metrics will help). >> In the last INFRA newsletter, they announced some limitations that will >> apply to the projects (effective as of end of April) - so once those will >> be followed, we should be "safe" from being impacted by others (i.e. >> noisy-neighbour effect). Some of the projects (not Airflow (!) ) were >> exceeding those so far and they will be capped - they will need to optimize >> their builds eventually. >> >> Those are the rules: >> >> * All workflows MUST have a job concurrency level less than or equal to >> 20. This means a workflow cannot have more than 20 jobs running at the same >> time across all matrices. >> * All workflows SHOULD have a job concurrency level less than or equal to >> 15. Just because 20 is the max, doesn't mean you should strive for 20. >> * The average number of minutes a project uses per calendar week MUST NOT >> exceed the equivalent of 25 full-time runners (250,000 minutes, or 4,200 >> hours). >> * The average number of minutes a project uses in any consecutive five-day >> period MUST NOT exceed the equivalent of 30 full-time runners (216,000 >> minutes, or 3,600 hours). >> * Projects whose builds consistently cross the maximum use limits will >> lose their access to GitHub Actions until they fix their build >> configurations. >> >> Those numbers on their own do not tell much, but we can easily see what >> they mean when we put them side-by-side t with "our" current numbers. >> >> * Currently - with all the "public" usage we are at 8 full-time runners. >> This is after some of the changes I've done, With the recent changes I >> already moved a lot of the non-essential build components that do not >> require a lot of parallelism to public runners. >> * The 20/15 jobs limit is a bit artificial (not really enforceable on >> workflow level) - but in our case as I optimized most PR to run just a >> subset of the tests, The average will be way below that - no matter if you >> are committer or not, regular PRs are far smaller subset of the jobs than >> full "canary" build. And for canary builds we should stay - at least for >> now - with self-hosted runners. >> >> Some of the back-of-the envelope calculations of what might happen when we >> switch to "public" for everyone: >> >> Unfortunately, until we enable the experiment, I do not have an easy way >> to distinguish the "canary" from "committer" runs so those are a bit >> guesses. But our self-hosted build time vs. public build time is ~ 20% more >> for self-hosted (100.000 minutes vs. 80.000 minutes this month) - see the >> attached screenshot for the current month. >> As you can see - building images are already moved to public runners for >> everyone as of two weeks or so, so that will not change. >> >> Taking into account that self-hosted ones are ~ 1.7x faster, this means >> that currently we have ~ 2x more self-hosted time used than public. We can >> assume that 50% of that are committer PRs and "Canary" builds are the >> second half (sounds safe because canary builds use way more resources, even >> if committers run many more PRs than merges). >> So by moving committer builds to public runners, we will - likely - >> increase our public time 2x (from 8 FT runners to 16 FT runners) - way >> below the 25 FT runners that is the "cap" from INFRA, Even if we move all >> Canary builds there, we should be at most at ~24 FTs, which is still below >> the limits. but would be dangerously close to it. That's why I want to keep >> canary builds as self-hosted until we can get some clarity on the "PR" >> moving impact. >> >> We will see the final numbers when we move, but I think we are pretty safe >> within the limits. >> >> J. >> >> >> On Fri, Apr 5, 2024 at 1:16 PM Hussein Awala <huss...@awala.fr<mailto: >> huss...@awala.fr>> wrote: >> Although 900 runners seem like a lot, they are shared among the Apache >> organization's 2.2k repositories, of course only a few of them are active >> (let's say 50), and some of them use an external CI tool for big jobs (eg: >> Kafka uses Jenkins, Hudi uses Azure pipelines), but we have other very >> active repositories based entirely on GHA, for example, Iceberg, Spark, >> Superset, ... >> >> I haven't found the AFS runners metrics dashboard to check the max >> concurrency and the max queued time during peak hours, but I'm sure that >> moving Airflow committers' CI jobs to public runners will put some pressure >> on these runners, especially since these committers are the most active >> contributors to Airflow, and the 35 self-hosted runners (with 8 CPUs and 64 >> GB RAM) are used almost all the time, so we can say that we will need >> around 70 AFS runners to run the same jobs. >> >> There is no harm in testing and deciding after 2-3 weeks. >> >> We also need to find a way to let the infra team help us solve the >> connectivity problem with the ARC runners >> < >> https://issues.apache.org/jira/projects/INFRA/issues/INFRA-25117?filter=reportedbyme >>> >> . >> >> +1 for testing what you propose. >> >> On Fri, Apr 5, 2024 at 12:07 PM Amogh Desai <amoghdesai....@gmail.com >> <mailto:amoghdesai....@gmail.com>> >> wrote: >> >>> +1 I like the idea. >>> Looking forward to seeing the difference. >>> >>> Thanks & Regards, >>> Amogh Desai >>> >>> >>> On Fri, Apr 5, 2024 at 3:54 AM Ferruzzi, Dennis >>> <ferru...@amazon.com.invalid> >>> wrote: >>> >>>> Interested in seeing the difference, +1 >>>> >>>> >>>> - ferruzzi >>>> >>>> >>>> ________________________________ >>>> From: Oliveira, Niko <oniko...@amazon.com.INVALID> >>>> Sent: Thursday, April 4, 2024 2:00 PM >>>> To: dev@airflow.apache.org<mailto:dev@airflow.apache.org> >>>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Consider disabling >>>> self-hosted runners for commiter PRs >>>> >>>> CAUTION: This email originated from outside of the organization. Do not >>>> click links or open attachments unless you can confirm the sender and >>> know >>>> the content is safe. >>>> >>>> >>>> >>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> externe. >>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne >>> pouvez >>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain >>> que >>>> le contenu ne présente aucun risque. >>>> >>>> >>>> >>>> +1I'd love to see this as well. >>>> >>>> In the past, stability and long queue times of PR builds have been very >>>> frustrating. I'm not 100% sure this is due to using self hosted >> runners, >>>> since 35 queue depth (to my mind) should be plenty. But something about >>>> that setup has never seemed quite right to me with queuing. Switching >> to >>>> public runners for a while to experiment would be great to see if it >>>> improves. >>>> >>>> ________________________________ >>>> From: Pankaj Koti <pankaj.k...@astronomer.io<mailto: >> pankaj.k...@astronomer.io>.INVALID> >>>> Sent: Thursday, April 4, 2024 12:41:02 PM >>>> To: dev@airflow.apache.org<mailto:dev@airflow.apache.org> >>>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Consider disabling >>>> self-hosted runners for commiter PRs >>>> >>>> CAUTION: This email originated from outside of the organization. Do not >>>> click links or open attachments unless you can confirm the sender and >>> know >>>> the content is safe. >>>> >>>> >>>> >>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> externe. >>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne >>> pouvez >>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain >>> que >>>> le contenu ne présente aucun risque. >>>> >>>> >>>> >>>> +1 from me to this idea. >>>> >>>> Sounds very reasonable to me. >>>> At times, my experience has been better with public runners instead of >>>> self-hosted runners :) >>>> >>>> And like already mentioned in the discussion, I think having the >> ability >>> of >>>> a applying the label "use-self-hosted-runners" to be used for critical >>>> times would be nice to have too. >>>> >>>> >>>> On Fri, 5 Apr 2024, 00:50 Jarek Potiuk, <ja...@potiuk.com<mailto: >> ja...@potiuk.com>> wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> TL;DR With some recent changes in GitHub Actions and the fact that >> ASF >>>> has >>>>> a lot of runners available donated for all the builds, I think we >> could >>>>> experiment with disabling "self-hosted" runners for committer builds. >>>>> >>>>> The self-hosted runners of ours have been extremely helpful (and we >>>> should >>>>> again thank Amazon and Astronomer for donating credits / money for >>>> those) - >>>>> when the Github Public runners have been far less powerful - and we >> had >>>>> less number of those available for ASF projects. This saved us a LOT >> of >>>>> troubles where there was a contention between ASF projects. >>>>> >>>>> But as of recently both limitations have been largely removed: >>>>> >>>>> * ASF has 900 public runners donated by GitHub to all projects >>>>> * Those public runners have (as of January) for open-source projects >>> now >>>>> have 4 CPUS and 16GB of memory - >>>>> >>>>> >>>> >>> >> https://github.blog/2024-01-17-github-hosted-runners-double-the-power-for-open-source/ >>>>> >>>>> >>>>> While they are not as powerful as our self-hosted runners, the >>>> parallelism >>>>> we utilise for those brings those builds in not-that bad shape >> compared >>>> to >>>>> self-hosted runners. Typical differences between the public and >>>> self-hosted >>>>> runners now for the complete set of tests are ~ 20m for public >> runners >>>> and >>>>> ~14 m for self-hosted ones. >>>>> >>>>> But this is not the only factor - I think committers experience the >>> "Job >>>>> failed" for self-hosted runners generally much more often than >>>>> non-committers (stability of our solution is not best, also we are >>> using >>>>> cheaper spot instances). Plus - we limit the total number of >>> self-hosted >>>>> runners (35) - so if several committers submit a few PRs and we have >>>> canary >>>>> build running, the jobs will wait until runners are available. >>>>> >>>>> And of course it costs the credits/money of sponsors which we could >> use >>>> for >>>>> other things. >>>>> >>>>> I have - as of recently - access to Github Actions metrics - and >> while >>>> ASF >>>>> is keeping an eye and stared limiting the number of parallel jobs >>>> workflows >>>>> in projects are run, it looks like even if all committer runs are >> added >>>> to >>>>> the public runners, we will still cause far lower usage that the >> limits >>>> are >>>>> and far lower than some other projects (which I will not name >> here). I >>>>> have access to the metrics so I can monitor our usage and react. >>>>> >>>>> I think possibly - if we switch committers to "public" runners by >>> default >>>>> -the experience will not be much worse for them (and sometimes even >>>> better >>>>> - because of stability/limited queue). >>>>> >>>>> I was planning this carefully - I made a number of refactors/changes >> to >>>> our >>>>> workflows recently that makes it way easier to manipulate the >>>> configuration >>>>> and get various conditions applied to various jobs - so >>>>> changing/experimenting with those settings should be - well - a >> breeze >>>> :). >>>>> Few recent changes had proven that this change and workflow refactor >>> were >>>>> definitely worth the effort, I feel like I finally got a control over >>> it >>>>> where previously it was a bit like herding a pack of cats (which I >>>>> brought to live by myself, but that's another story). >>>>> >>>>> I would like to propose to run an experiment and see how it works if >> we >>>>> switch committer PRs back to the public runners - leaving the >>> self-hosted >>>>> runners only for canary builds (which makes perfect sense because >> those >>>>> builds run a full set of tests and we need as much speed and power >>> there >>>> as >>>>> we can. >>>>> >>>>> This is pretty safe, We should be able to switch back very easily if >> we >>>> see >>>>> problems. I will also monitor it and see if our usage is within the >>>> limits >>>>> of the ASF. I can also add the feature that committers should be able >>> to >>>>> use self-hosted runners by applying the "use self-hosted runners" >> label >>>> to >>>>> a PR. >>>>> >>>>> Running it for 2-3 weeks should be enough to gather experience from >>>>> committers - whether things will seem better or worse for them - or >>> maybe >>>>> they won't really notice a big difference. >>>>> >>>>> Later we could consider some next steps - disabling the self-hosted >>>> runners >>>>> for canary builds if we see that our usage is low and build are fast >>>>> enough, eventually possibly removing current self-hosted runners and >>>>> switching to a better k8s based infrastructure (which we are close to >>> do >>>>> but it makes it a bit difficult while current self-hosted solution is >>> so >>>>> critical to keep it running (like rebuilding the plane while it is >>>> flying). >>>>> I'd love to do it gradually in the "change slowly and observe" mode - >>>>> especially now that I have access to "proper" metrics. >>>>> >>>>> WDYT? >>>>> >>>>> J. >>>>> >>>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
