On behalf of the Apache DataFusion and Apache Arrow (Rust) projects, we very much appreciate the work that has gone into reducing the amount of time that the Airflow project (and other projects) uses on the shared github runners.
Last week, as we were preparing a bunch of release candidates, we regularly saw CI jobs wait 4-7 hours to start; having to rerun a job (e.g. due to resolving merge conflicts) would often add an extra day of merge latency. This week the wait time is more like 30-60 minutes. I think all projects are struggling with the impacts and changes from AI agents, and I am very happy that we were able to make progress via collaboration between projects, without having to have some sort of ASF wide mandate. Thank you especially to Jarek for helping us understand what was going on, and for his efforts to keep us informed. Finally, for what it is worth, I think our experience so far with our "Run CI for external contributors on their own forks" has been very positive. Contributor CI runs faster, and we don't slow down our own merge queue. Thank you again, Andrew On Mon, Oct 26, 2026 at 5:58 PM Jarek Potiuk <[email protected]> wrote: > Ash, ( +Andrew), > > Lower CI usage has always been mentioned as a side effect. And you even > explained that you "don't care about this side effect". > > Always. Look at the discussion. But it will, as I explained, limit the > throughput of big PRs contributors will open. See this PR: > https://github.com/apache/airflow/pull/75695#issue-6047731044 which is > the first "full test" PR that runs after small fixes I merged today - so it > is ready to start the pilot. It is now 48 minutes in, and it looks like it > is ~ 20% done - which means it will take 4 hours to complete and they would > not be able to run any other Airflow PR jobs. > > And, we have 4 external contributors absolutely eager to volunteer and > test it, helping the whole community - including the ASF community—deal > with the AI-generated load. They seem very concerned about our community > health - including us maintainers and the whole ASF community—that suffers > from it, to the extent that they accept CI will be working from their > forks, potentially slowing them down on big prs. > > PR is here https://github.com/apache/airflow/pull/75696 ., I would > appreciate a review. > > And you can see this discussion opened in Airflow by Andrew Lamb—whom I > did not treat very well when he started nudging us daily (sorry Andrew): > https://github.com/apache/airflow/discussions/75395 for example. > > You can also see the discussion where I worried that such constant nagging > would stir up some "initial objectors" running queues. You can see the > discussion here - where I also found, with Magpie new skill - 30% - 40% > savings for Arrow (even though it turns out that they already decreased > their usage heavily during the last weeks. And Andrew was one of the people > working on it. > > You might find it telling, that I asked Andrew to stop nagging us - > because of You. Yes I upset other people so they would not accidentally > upset you. This is how far I go to protect against your anger bursts, Ash. > > It wasn't because he did not notice LAZY CONSENSUS, but because I was > afraid that if you noticed the nagging from other projects who would expect > us to do something, you might say "I don't care" and break the LAZY > CONSENSUS in some kind of uncontrolled anger burst - that has no rational > basis. > > Also, there was lots of discussion last week in the ASF Slack (available > for ASF members - for example you, Ash). > > * #builds https://the-asf.slack.com/archives/CF6PY5M4N > * #project-workflow-optimisation - > https://the-asf.slack.com/archives/C0B835NQG6P > > In those threads we discussed all the different ways we can help ASF, and > switching to CI from forks was named by Andrew as the "single biggest > saving" they achieved in Arrow. And I had to literally defend the proposal > to heavlily limit Airflow in how much jobs it can run - because at some > point of time (when we did Airflow 3.4.0 preparation + regular PRs), we > took 30% of the ASF runners' capacity. > > You can also see this datafusion issue about it: > https://github.com/apache/datafusion/issues/25088#issuecomment-6419032370 > > Also, you can see the result of that dicussion shared action submitted by > Datafusion maintainer > https://github.com/apache/infrastructure-actions/pull/1444 - Datafusion > **also** implemented "CI from forks" - to decrease the queue for eveyrone. > And I am going to get it merged, improved with what we learn in Airflow and > reach out to **all** PMCs who have big usage to consider switching to it. > Even more Magpie skill will help them to do manually. > > All this happened last week. And I posted about it also in > https://lists.apache.org/thread.html/kcxdn26wpnlwonq0tf09bg0lykpr7oc3 > including > adding Arrow and Datafusion devlist. > > I suggest we focus on fine-tuning it now. You focus on making sure task > loop works, Shahar and myself and a number of other people who help and > have **no problem** with that change or paid more attaention - on that, > > For the benefit of everyone. > > J > > On Mon, Oct 26, 2026 at 8:50 PM Ash Berlin-Taylor <[email protected]> wrote: > >> Thanks Holden, that seems to be Arrow though, not Airflow? >> >> > On 26 Oct 2026, at 19:42, Holden Karau <[email protected]> wrote: >> > >> > So a number of ASF projects have been called on the reduce CI load >> > (including Airflow and Spark). There's been a number of e-mails about >> this, >> > one of the airflow e-mails is >> > >> https://lists.apache.org/thread.html/0cs9lo9lqxcdbzpqt1yhzvzx5pn1h7dt?find_parent=true >> > >> > On Mon, Oct 26, 2026 at 12:20 PM Ash Berlin-Taylor <[email protected]> >> wrote: >> > >> >> First off: If a proposal is big enough to need an AIP, definitionally, >> it >> >> is not appropriate for lazy consensus. AIP’s are for big changes. Big >> >> changes get actively voted upon. And since it _does_ have an AIP, you >> >> already acknowledged at some level it is a big change. >> >> >> >> AIP-120 starts out by saying "Apache Airflow receives more pull >> requests >> >> than its maintainers can review”. That framing is very different to >> “ASF >> >> org asked us to reduce our CI usage”. >> >> >> >> Where in the AIP document or previous discussion thread was the “we >> need >> >> to reduce our CI usage” called out as a goal? Ci time is mentioned, but >> >> never “we need to reduce this because the ASF org has asked us to”. I >> too >> >> would like my CI to jobs to run quicker. Who wouldn’t? >> >> >> >> Where have we been called upon to reduce CI load? It was on this list? >> I >> >> can’t see it. And if it isn’t on list it essentially didn’t happen. >> Another >> >> ASF maxim, of which you are also painfully aware. >> >> >> >> I feel you are taking liberties here and making big changes to the >> project >> >> almost unilaterally. And I think you should apologise for doing it in a >> >> tone deaf way without true consensus. >> >> >> >> >> >> >> >>> On 26 Oct 2026, at 18:29, Jarek Potiuk <[email protected]> wrote: >> >>> >> >>> BTW. Airflow has been called upon many times last week by other >> projects >> >> as >> >>> *biggest resource hog" many times, people were angry, people asked >> Infra >> >>> to limit Airflow's resource usage. They literally called Infra to >> shut us >> >>> down during the release process. >> >>> >> >>> The only way I contained that was by telling people we are already >> >> working >> >>> on it and we have a clear plan to follow. This required a lot of >> "being >> >>> firm". One comment I received was that after 3 weeks of effort, I >> finally >> >>> refined the design enough that it raised no objections, allowing me to >> >>> follow through and help other projects—including helping them reduce >> >> their >> >>> usage - which benefits everyone. >> >>> >> >>> I added this page >> >> https://infra.apache.org/github-actions-optimization.html >> >>> and added a feature in Magpie that allows people to optimize their >> >>> workflows practically automatically.. >> >>> >> >>> All that following all the rules we agreed to. >> >>> >> >>> I will leave it at that. >> >>> >> >>> J. >> >>> >> >>> >> >>> On Mon, Oct 26, 2026 at 7:22 PM Jarek Potiuk <[email protected]> >> wrote: >> >>> >> >>>> Sorry. We reduced our resource usage by 2/3 >> >>>> >> >>>> On Mon, Oct 26, 2026 at 7:20 PM Jarek Potiuk <[email protected]> >> wrote: >> >>>> >> >>>>> Ash, >> >>>>> >> >>>>> * The original discussion where I posted my last response from me to >> >> you >> >>>>> on the change was Sep 22 and waited for your answer. That was 1.5 >> >> months >> >>>>> ago. >> >>>>> >> https://lists.apache.org/thread.html/w74655o8y1jzrzcnygntyyovocf7zmzj >> >> - >> >>>>> I have not seen you respond to it and I believe I answered your >> >> concerns >> >>>>> >> >>>>> *.I implemented PR limits as the first step (with your consent) of >> >>>>> improvements and announced my plan for the next steps >> >>>>> >> https://lists.apache.org/thread.html/y0vzry5s55k2gy17rc93632zp0d5733o >> >>>>> >> >>>>> * I adapted design, respondsed to your concerns and explained what >> I am >> >>>>> going to do on this thread - where I started DISCUSSION on the >> updated >> >>>>> design - >> >>>>> >> https://lists.apache.org/thread.html/ky392mrrcmj312y1r276r30p584n804c >> >> - >> >>>>> that was 21 days ago. - it did not receive any objection from you >> >>>>> >> >>>>> * the discussion run until - 10 days ago >> >>>>> >> https://lists.apache.org/thread.html/ky392mrrcmj312y1r276r30p584n804c >> >> - >> >>>>> where no-one raised significant objections, this discussion had >> links >> >> to >> >>>>> design, AIP, Draft PRs and my clear intentions to run LAZY CONSENSUS >> >>>>> >> >>>>> * I raised Lazy consensus >> >>>>> >> https://lists.apache.org/thread.html/nfno4r1jqx5dj75v867prjjgg6o77odx >> >> - >> >>>>> 7 days ago >> >>>>> >> >>>>> * I announced result on Friday: - 4 days after it has been waiting >> >>>>> >> >>>>> Yes. AIP-120 is approved. I cannot account for someone not paying >> >>>>> attention for 1.5 months, sorry. You are just one of the PMC members >> >> here - >> >>>>> nobody else does not object. You are not more important than others >> >> and we >> >>>>> are not going to wait with decisions, that also help the whole ASF >> deal >> >>>>> with the difficult situation of CI resources being exhausted by >> >>>>> AI—especially since we gave ample time to respond to two discussions >> >> and >> >>>>> the lazy consensus. >> >>>>> >> >>>>> Get over that, Ash. If you would like to change somethign in the way >> >> how >> >>>>> AIP-120 is configured feel free to start discussiosn (you already >> did >> >> in >> >>>>> #contributors and made some suggestions - which is good). >> >>>>> >> >>>>> It is implemented as is now. We can disable it any time if we do not >> >> want >> >>>>> to lose our contributors. We have many configuration options, and we >> >>>>> already reduced our resource usage by one-third, unblocking not only >> >> our >> >>>>> queue but also other projects' queues. >> >>>>> >> >>>>> Let's discuss what we want to change and how to configure it, >> simply. >> >>>>> Let's work toward the common goal of faster and better CI for >> Airflow. >> >>>>> >> >>>>> J. >> >>>>> >> >>>>> >> >>>>> On Mon, Oct 26, 2026 at 6:44 PM Ash Berlin-Taylor <[email protected]> >> >> wrote: >> >>>>> >> >>>>>> No. >> >>>>>> >> >>>>>> Jarek: this behaviour _was not okay_. You knew when creating the >> lazy >> >>>>>> consensus that there was not consensus. >> >>>>>> >> >>>>>> This was never a valid use case of a Lazy Consensus. This >> discussion >> >>>>>> already had strong disagreements. I was the most vocal, but I >> wasn’t >> >> the >> >>>>>> only person to mention disagreement. And yet you really thought >> lazy >> >>>>>> consensus was appropriate tool here? How would you feel if I >> passed a >> >>>>>> “Remove Breeze” via lazy consensus when I knew you were on holiday? >> >>>>>> >> >>>>>> As per ASF’s own guidance, which I know you already know. >> >>>>>> >> >>>>>>> Sometimes lazy consensus is not appropriate. In such cases it is >> >>>>>> necessary to make a proposal to the email list and discuss options >> >>>>>> >> >>>>>> And since the discussion _already_ had pushback, lazy consensus was >> >>>>>> categorically not appropriate. >> >>>>>> >> >>>>>> This is not an appropriate response Jarek. Not even remotely. If >> you >> >>>>>> think it is you need to re-evaluate your priorities around >> community >> >>>>>> building. >> >>>>>> >> >>>>>> I repeat. There is no consensus on AIP-120. >> >>>>>> >> >>>>>>> On 26 Oct 2026, at 14:59, Jarek Potiuk <[email protected]> wrote: >> >>>>>>> >> >>>>>>>> There is no consensus reached here. >> >>>>>>> >> >>>>>>> Consensus has been reached. We've been through that before; there >> >> was a >> >>>>>>> long discussion followed by arguments from the other side. Your >> only >> >>>>>>> opposition was that it is aimed only to save CI time of shared >> >> runners >> >>>>>>> (which also it does). >> >>>>>>> And as the last time we discussed - you can always call to revert >> >> that >> >>>>>>> decision. AIP-120 was approved 3 days ago. >> >>>>>>> >> >>>>>>>> Calling a lazy consensus on this AIP given how strongly I voiced >> my >> >>>>>>> opposition to it is appropriate. >> >>>>>>> >> >>>>>>> You can always call a change to remove it. In the meantime, we >> >> proceed >> >>>>>> with >> >>>>>>> testing it. Also, an alternative is that we would likely have to >> cut >> >>>>>> down >> >>>>>>> our usage of shared ASF CI because we are taking nearly 30% of it. >> >>>>>> Some of >> >>>>>>> the changes and optimizations Shahar and I implemented over the >> >> weekend >> >>>>>>> reduced our usage to one-third (also Arrow and Datafusion also >> made >> >>>>>> similar >> >>>>>>> optimizations with room for more) - which already had an effect on >> >>>>>>> significant decrease of queue time (which was also goal of >> AIP-120). >> >>>>>>> >> >>>>>>> But yes - feel free to start a new discussion on reverting it. >> This >> >>>>>> is the >> >>>>>>> way how decisions are being made here. >> >>>>>>> >> >>>>>>> J >> >>>>>>> >> >>>>>>> >> >>>>>>> On Mon, Oct 26, 2026 at 2:23 PM Ash Berlin-Taylor <[email protected] >> > >> >>>>>> wrote: >> >>>>>>> >> >>>>>>>> I’m angry and making typos. Is _not appropriate_. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>>> On 26 Oct 2026, at 13:22, Ash Berlin-Taylor <[email protected]> >> >> wrote: >> >>>>>>>>> >> >>>>>>>>> Calling a lazy consensus on this AIP given how strongly I >> voiced my >> >>>>>>>> opposition to it is appropriate. >> >>>>>>>>> >> >>>>>>>>> This should have never a candidate for lazy consensus, and was >> >> opened >> >>>>>>>> and closed when many of us were busy working on the 3.4 release. >> >>>>>> There is >> >>>>>>>> no consensus reached here. >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>> On 23 Oct 2026, at 21:52, Jarek Potiuk <[email protected]> >> wrote: >> >>>>>>>>>> >> >>>>>>>>>> Hello everyone, >> >>>>>>>>>> >> >>>>>>>>>> The lazy consensus on AIP-120 [1] ended on 2026-10-22 12:00 UTC >> >> with >> >>>>>>>>>> no objections, so AIP-120 is accepted. >> >>>>>>>>>> >> >>>>>>>>>> Next steps, as described in the proposal: merge the design PR >> and >> >>>>>> the >> >>>>>>>>>> stacked implementation PRs (everything ships switched off), run >> >> the >> >>>>>>>>>> opt-in pilot, then switch it on for all external contributors >> in a >> >>>>>>>>>> separate one-line change. >> >>>>>>>>>> >> >>>>>>>>>> Reviews on the stack are very welcome: #73378, #74819, #74820, >> >>>>>> #74822, >> >>>>>>>>>> #74823, #74825. >> >>>>>>>>>> >> >>>>>>>>>> [1] >> >>>>>> https://lists.apache.org/thread/nfno4r1jqx5dj75v867prjjgg6o77odx >> >>>>>>>>>> >> >>>>>>>>>> J. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>> >> --------------------------------------------------------------------- >> >>>>>>>>>> To unsubscribe, e-mail: [email protected] >> >>>>>>>>>> For additional commands, e-mail: [email protected] >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >> --------------------------------------------------------------------- >> >>>>>>>>> To unsubscribe, e-mail: [email protected] >> >>>>>>>>> For additional commands, e-mail: [email protected] >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >> --------------------------------------------------------------------- >> >>>>>>>> To unsubscribe, e-mail: [email protected] >> >>>>>>>> For additional commands, e-mail: [email protected] >> >>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> --------------------------------------------------------------------- >> >>>>>> To unsubscribe, e-mail: [email protected] >> >>>>>> For additional commands, e-mail: [email protected] >> >>>>>> >> >>>>>> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: [email protected] >> >> For additional commands, e-mail: [email protected] >> >> >> >> >> > >> > -- >> > Twitter: https://twitter.com/holdenkarau >> > Fight Health Insurance: https://www.fighthealthinsurance.com/ >> > <https://www.fighthealthinsurance.com/?q=hk_email> >> > Books (Learning Spark, High Performance Spark, etc.): >> > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> > Pronouns: she/her >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >>
