Ah my bad, got my projects mixed up. It’s possible Airlfow wasn’t directly asked and it’s just a number of other projects were asked to cut back.
Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her On Mon, Oct 26, 2026 at 12:50 PM Ash Berlin-Taylor <[email protected]> wrote: > Thanks Holden, that seems to be Arrow though, not Airflow? > > > On 26 Oct 2026, at 19:42, Holden Karau <[email protected]> wrote: > > > > So a number of ASF projects have been called on the reduce CI load > > (including Airflow and Spark). There's been a number of e-mails about > this, > > one of the airflow e-mails is > > > https://lists.apache.org/thread.html/0cs9lo9lqxcdbzpqt1yhzvzx5pn1h7dt?find_parent=true > > > > On Mon, Oct 26, 2026 at 12:20 PM Ash Berlin-Taylor <[email protected]> > wrote: > > > >> First off: If a proposal is big enough to need an AIP, definitionally, > it > >> is not appropriate for lazy consensus. AIP’s are for big changes. Big > >> changes get actively voted upon. And since it _does_ have an AIP, you > >> already acknowledged at some level it is a big change. > >> > >> AIP-120 starts out by saying "Apache Airflow receives more pull requests > >> than its maintainers can review”. That framing is very different to “ASF > >> org asked us to reduce our CI usage”. > >> > >> Where in the AIP document or previous discussion thread was the “we need > >> to reduce our CI usage” called out as a goal? Ci time is mentioned, but > >> never “we need to reduce this because the ASF org has asked us to”. I > too > >> would like my CI to jobs to run quicker. Who wouldn’t? > >> > >> Where have we been called upon to reduce CI load? It was on this list? I > >> can’t see it. And if it isn’t on list it essentially didn’t happen. > Another > >> ASF maxim, of which you are also painfully aware. > >> > >> I feel you are taking liberties here and making big changes to the > project > >> almost unilaterally. And I think you should apologise for doing it in a > >> tone deaf way without true consensus. > >> > >> > >> > >>> On 26 Oct 2026, at 18:29, Jarek Potiuk <[email protected]> wrote: > >>> > >>> BTW. Airflow has been called upon many times last week by other > projects > >> as > >>> *biggest resource hog" many times, people were angry, people asked > Infra > >>> to limit Airflow's resource usage. They literally called Infra to shut > us > >>> down during the release process. > >>> > >>> The only way I contained that was by telling people we are already > >> working > >>> on it and we have a clear plan to follow. This required a lot of "being > >>> firm". One comment I received was that after 3 weeks of effort, I > finally > >>> refined the design enough that it raised no objections, allowing me to > >>> follow through and help other projects—including helping them reduce > >> their > >>> usage - which benefits everyone. > >>> > >>> I added this page > >> https://infra.apache.org/github-actions-optimization.html > >>> and added a feature in Magpie that allows people to optimize their > >>> workflows practically automatically.. > >>> > >>> All that following all the rules we agreed to. > >>> > >>> I will leave it at that. > >>> > >>> J. > >>> > >>> > >>> On Mon, Oct 26, 2026 at 7:22 PM Jarek Potiuk <[email protected]> wrote: > >>> > >>>> Sorry. We reduced our resource usage by 2/3 > >>>> > >>>> On Mon, Oct 26, 2026 at 7:20 PM Jarek Potiuk <[email protected]> > wrote: > >>>> > >>>>> Ash, > >>>>> > >>>>> * The original discussion where I posted my last response from me to > >> you > >>>>> on the change was Sep 22 and waited for your answer. That was 1.5 > >> months > >>>>> ago. > >>>>> > https://lists.apache.org/thread.html/w74655o8y1jzrzcnygntyyovocf7zmzj > >> - > >>>>> I have not seen you respond to it and I believe I answered your > >> concerns > >>>>> > >>>>> *.I implemented PR limits as the first step (with your consent) of > >>>>> improvements and announced my plan for the next steps > >>>>> > https://lists.apache.org/thread.html/y0vzry5s55k2gy17rc93632zp0d5733o > >>>>> > >>>>> * I adapted design, respondsed to your concerns and explained what I > am > >>>>> going to do on this thread - where I started DISCUSSION on the > updated > >>>>> design - > >>>>> > https://lists.apache.org/thread.html/ky392mrrcmj312y1r276r30p584n804c > >> - > >>>>> that was 21 days ago. - it did not receive any objection from you > >>>>> > >>>>> * the discussion run until - 10 days ago > >>>>> > https://lists.apache.org/thread.html/ky392mrrcmj312y1r276r30p584n804c > >> - > >>>>> where no-one raised significant objections, this discussion had links > >> to > >>>>> design, AIP, Draft PRs and my clear intentions to run LAZY CONSENSUS > >>>>> > >>>>> * I raised Lazy consensus > >>>>> > https://lists.apache.org/thread.html/nfno4r1jqx5dj75v867prjjgg6o77odx > >> - > >>>>> 7 days ago > >>>>> > >>>>> * I announced result on Friday: - 4 days after it has been waiting > >>>>> > >>>>> Yes. AIP-120 is approved. I cannot account for someone not paying > >>>>> attention for 1.5 months, sorry. You are just one of the PMC members > >> here - > >>>>> nobody else does not object. You are not more important than others > >> and we > >>>>> are not going to wait with decisions, that also help the whole ASF > deal > >>>>> with the difficult situation of CI resources being exhausted by > >>>>> AI—especially since we gave ample time to respond to two discussions > >> and > >>>>> the lazy consensus. > >>>>> > >>>>> Get over that, Ash. If you would like to change somethign in the way > >> how > >>>>> AIP-120 is configured feel free to start discussiosn (you already did > >> in > >>>>> #contributors and made some suggestions - which is good). > >>>>> > >>>>> It is implemented as is now. We can disable it any time if we do not > >> want > >>>>> to lose our contributors. We have many configuration options, and we > >>>>> already reduced our resource usage by one-third, unblocking not only > >> our > >>>>> queue but also other projects' queues. > >>>>> > >>>>> Let's discuss what we want to change and how to configure it, simply. > >>>>> Let's work toward the common goal of faster and better CI for > Airflow. > >>>>> > >>>>> J. > >>>>> > >>>>> > >>>>> On Mon, Oct 26, 2026 at 6:44 PM Ash Berlin-Taylor <[email protected]> > >> wrote: > >>>>> > >>>>>> No. > >>>>>> > >>>>>> Jarek: this behaviour _was not okay_. You knew when creating the > lazy > >>>>>> consensus that there was not consensus. > >>>>>> > >>>>>> This was never a valid use case of a Lazy Consensus. This discussion > >>>>>> already had strong disagreements. I was the most vocal, but I wasn’t > >> the > >>>>>> only person to mention disagreement. And yet you really thought > lazy > >>>>>> consensus was appropriate tool here? How would you feel if I passed > a > >>>>>> “Remove Breeze” via lazy consensus when I knew you were on holiday? > >>>>>> > >>>>>> As per ASF’s own guidance, which I know you already know. > >>>>>> > >>>>>>> Sometimes lazy consensus is not appropriate. In such cases it is > >>>>>> necessary to make a proposal to the email list and discuss options > >>>>>> > >>>>>> And since the discussion _already_ had pushback, lazy consensus was > >>>>>> categorically not appropriate. > >>>>>> > >>>>>> This is not an appropriate response Jarek. Not even remotely. If you > >>>>>> think it is you need to re-evaluate your priorities around community > >>>>>> building. > >>>>>> > >>>>>> I repeat. There is no consensus on AIP-120. > >>>>>> > >>>>>>> On 26 Oct 2026, at 14:59, Jarek Potiuk <[email protected]> wrote: > >>>>>>> > >>>>>>>> There is no consensus reached here. > >>>>>>> > >>>>>>> Consensus has been reached. We've been through that before; there > >> was a > >>>>>>> long discussion followed by arguments from the other side. Your > only > >>>>>>> opposition was that it is aimed only to save CI time of shared > >> runners > >>>>>>> (which also it does). > >>>>>>> And as the last time we discussed - you can always call to revert > >> that > >>>>>>> decision. AIP-120 was approved 3 days ago. > >>>>>>> > >>>>>>>> Calling a lazy consensus on this AIP given how strongly I voiced > my > >>>>>>> opposition to it is appropriate. > >>>>>>> > >>>>>>> You can always call a change to remove it. In the meantime, we > >> proceed > >>>>>> with > >>>>>>> testing it. Also, an alternative is that we would likely have to > cut > >>>>>> down > >>>>>>> our usage of shared ASF CI because we are taking nearly 30% of it. > >>>>>> Some of > >>>>>>> the changes and optimizations Shahar and I implemented over the > >> weekend > >>>>>>> reduced our usage to one-third (also Arrow and Datafusion also made > >>>>>> similar > >>>>>>> optimizations with room for more) - which already had an effect on > >>>>>>> significant decrease of queue time (which was also goal of > AIP-120). > >>>>>>> > >>>>>>> But yes - feel free to start a new discussion on reverting it. > This > >>>>>> is the > >>>>>>> way how decisions are being made here. > >>>>>>> > >>>>>>> J > >>>>>>> > >>>>>>> > >>>>>>> On Mon, Oct 26, 2026 at 2:23 PM Ash Berlin-Taylor <[email protected]> > >>>>>> wrote: > >>>>>>> > >>>>>>>> I’m angry and making typos. Is _not appropriate_. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> On 26 Oct 2026, at 13:22, Ash Berlin-Taylor <[email protected]> > >> wrote: > >>>>>>>>> > >>>>>>>>> Calling a lazy consensus on this AIP given how strongly I voiced > my > >>>>>>>> opposition to it is appropriate. > >>>>>>>>> > >>>>>>>>> This should have never a candidate for lazy consensus, and was > >> opened > >>>>>>>> and closed when many of us were busy working on the 3.4 release. > >>>>>> There is > >>>>>>>> no consensus reached here. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> On 23 Oct 2026, at 21:52, Jarek Potiuk <[email protected]> > wrote: > >>>>>>>>>> > >>>>>>>>>> Hello everyone, > >>>>>>>>>> > >>>>>>>>>> The lazy consensus on AIP-120 [1] ended on 2026-10-22 12:00 UTC > >> with > >>>>>>>>>> no objections, so AIP-120 is accepted. > >>>>>>>>>> > >>>>>>>>>> Next steps, as described in the proposal: merge the design PR > and > >>>>>> the > >>>>>>>>>> stacked implementation PRs (everything ships switched off), run > >> the > >>>>>>>>>> opt-in pilot, then switch it on for all external contributors > in a > >>>>>>>>>> separate one-line change. > >>>>>>>>>> > >>>>>>>>>> Reviews on the stack are very welcome: #73378, #74819, #74820, > >>>>>> #74822, > >>>>>>>>>> #74823, #74825. > >>>>>>>>>> > >>>>>>>>>> [1] > >>>>>> https://lists.apache.org/thread/nfno4r1jqx5dj75v867prjjgg6o77odx > >>>>>>>>>> > >>>>>>>>>> J. > >>>>>>>>>> > >>>>>>>>>> > >>>>>> > --------------------------------------------------------------------- > >>>>>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>>>>> For additional commands, e-mail: [email protected] > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >> --------------------------------------------------------------------- > >>>>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>>>> For additional commands, e-mail: [email protected] > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >> --------------------------------------------------------------------- > >>>>>>>> To unsubscribe, e-mail: [email protected] > >>>>>>>> For additional commands, e-mail: [email protected] > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> > --------------------------------------------------------------------- > >>>>>> To unsubscribe, e-mail: [email protected] > >>>>>> For additional commands, e-mail: [email protected] > >>>>>> > >>>>>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > > > -- > > Twitter: https://twitter.com/holdenkarau > > Fight Health Insurance: https://www.fighthealthinsurance.com/ > > <https://www.fighthealthinsurance.com/?q=hk_email> > > Books (Learning Spark, High Performance Spark, etc.): > > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > > Pronouns: she/her > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
