Vlada, Michał, Amazon team - you are the most "advanced" in system tests.
Would that proposed workflow be something that you'd see "good". We can also work on better notifications along the way—for example, by having dedicated Slack channels where new approvals show up, one per steward, to make it easier to know when decisions are pending. Other ideas are of course most welcome. J. On Fri, Oct 9, 2026 at 4:37 PM Jarek Potiuk <[email protected]> wrote: > > I'd like to make sure that the proposal on the table is not expecting > maintainers to do that. > > We discussed this with Shubham (and this is also included in > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451979463/WIP+AIP-124+Airflow-Powered+Agentic+Workflows+for+Project+Operations > ). A number of features we have in Airflow can now be used to our > advantage to build simple-to-manage workflows. > > My proposal (which includes the ability to control this via the Airflow > UI): > > 1) The system tests will not run by default on all PRs (to avoid abuse) - > what will happen is that if a PR will be determined to "expect" system > tests, it will have a "Waiting" status that will block it from merging (and > the system tests workflows will start - but **pause** on HITL approving > it). > > 2) Stewards (with multi-team support, we will be able to do this > selectively - different stewards for different providers) will be able to > say "Go ahead" via HITL, which will make the workflow continue. No > maintainer decision is needed yet. Of course Maintainers could also do this. > > 3) This might be set to "sticky"—meaning the HITL prompt won't occur for > re-runs, updates, or fixups, especially for somewhat trusted committers. > Stakeholders might choose this option (note that the primary risk involves > abusing steward-provided credits). We can also use deterministic and > AI-driven checks to determine if incoming changes are "dangerous" and make > different decisions even on a "sticky" PR. > > 4) Maintainers might only get involved when all checks - including the > "System Test" check is green. > > This has few nice properties: > > * it puts responsibility of triggering the System Tests on the Steward - > and until they do that, the PR will never get green (unless forced of > course) - we should enable overriding the state by maintainers for some > edge cases (no - we do not want to run system tests here) > * it starts contributor -> steward relationship happening early (we need > to surface the information who is the steward - wich we already planned) > * maintainer gets the PR to review when it's green - **including system > tests** > > We can configure it selectively, per provider. Steward will have to be > involved - both to provide the credits to run the tests, and to review the > changes and approve the run (with the sticky bit). > > I've been running similar workflows (without Airflow and triggered with > Magpie via CLI so far - for infrastructure-actions in ASF) - > https://github.com/apache/infrastructure-actions -> where we accept > potentially dangerous actions submitted to ASF - and I got **really** > comprehensive detetministic verifier of dangerous patterns + AI reviews > after that give **very** high confidence that we can run such tests on our > infrastructure. > > This is what has been enabled by AI agents. > > J. > > > > On Fri, Oct 9, 2026 at 2:07 PM Elad Kalif <[email protected]> wrote: > >> Hello, >> >> I'd like to raise a concern around: >> >> > We understand that not everyone has an active GCP environment. On the >> > Google/maintainer side, we’re looking into ways to make it easier to >> > trigger specific existing system tests directly on PR branches via CI >> for >> > trusted contributors. Until that workflow is ready, the responsibility >> of >> > proving a change works has to sit with the author. >> >> The responsibility is on the author but I'd like to clarify. Users often >> ping maintainers to review/merge their code. >> If only mantainers have the ability to run the system tests then PR >> authors >> have no way to validate a fix without mantainers rerunning the tests over >> and over. >> I'd like to make sure that the proposal on the table is not expecting >> maintainers to do that. >> >> On Fri, Oct 9, 2026 at 12:25 AM Jarek Potiuk <[email protected]> wrote: >> >> > I sugest let's just focus on the merit and solving problems. >> > >> > I proposed some next steps that are very well aligned with what we >> already >> > planned. >> > >> > >> > I believe we will see an AIP proposal tomorrow from Shubham that will >> > already incorporate the idea of using our own Airflow - not only for >> > automation of handling security issues and PRs but also to make it >> possible >> > to self service our provider's stewards and implement tolling that we >> had >> > already agreed to in AIP-85. With Google being first case to test it >> with. >> > Possibly even Amazon can join that trial. >> > >> > Where our stewards will be able to not only donate their credits but >> also >> > self-manage the System Test execution and where we will integrate it >> with >> > our CI and gating PRs on additional harness. >> > >> > The problem that Vlada raised is real, and it is affecting our users. >> And - >> > as Ash asked - I am sure Google will cover GCP costs involved (as >> promised >> > and discussed in the past). And it also will help us to manage AI >> overload, >> > and allow us dogfood Airflow and promote it's AI native workflow >> readiness. >> > >> > For me it is no brainer that we should try it. If anyone has ay other >> ideas >> > how we can address any of those problems and challenges we have - it's a >> > good idea to propose it. >> > >> > Let's focus on that. >> > >> > J. >> > >> > On Thu, Oct 8, 2026, 20:45 Ash Berlin-Taylor <[email protected]> wrote: >> > >> > > Sorry Vlada, I do owe you an apology. Sorry for the offence. >> > > >> > > My reading of this original message: is an unknown name*, that has >> sent >> > no >> > > previous emails >> > > >> https://lists.apache.org/[email protected]:gte%3D0d:Ulada >> > , >> > > and the email content saying >> > > >> > > > We want to agree on a clear baseline for PRs touching functional or >> > auth >> > > logic in the Google Provider; <3 things _YOU MUST DO_>.” >> > > >> > > What conculsion am I ment to draw form that, when this person has >> never >> > > sent anything before. And. More crucially, doesn’t give _any_ >> indication >> > > that “here is what I think we should do” until the very last line. >> > > >> > > *I have now been informed that it is a different spelling of Vlada >> (thank >> > > you Jarek), which would every so slightly change my point. This is >> still >> > > presented as fait-a-compli in everywhere apart from the subject line. >> > > _That_ is my main complaint. >> > > >> > > >> > > As for not working at google: And I am meant to just know that fact >> from >> > > the google.com <http://google.com/> email address? >> > > >> > > Jarek: “I didn’t find anything rude in the message”.. Saying “I didn’t >> > > find it rude” is another way of saying “my [Ash’s] opinion doesn’t >> > > matter”. Please do not ever. EVER. do that again, it is even more rude >> > and >> > > gaslighting than you think I was. What I heard when you said that: >> “That >> > > opinion you have: yeah it doesn’t matter. My opinion is more >> important.”. >> > > >> > > >> > > >> > > >> Also just to remind you Ash and everyone. Google is paying me >> **every >> > > >> single month** - similarly as Astronomer so that I can spend whole >> my >> > > time >> > > >> > > Any? That is relevant to the 100s of other people contributing every >> > month >> > > in what way? Or to anyone who wants to contribute a change to the >> google >> > > provider? >> > > >> > > >> > > > On 8 Oct 2026, at 16:56, Michał Modras via dev < >> [email protected] >> > > >> > > wrote: >> > > > >> > > > Hi Ash, >> > > > >> > > > You have really, really missed the mark in the tone you used in your >> > > public >> > > > criticism of a fellow Airflow community member who across years >> > > contributed >> > > > dozens of fixes, features, system tests and ongoing provider >> > stewardship. >> > > > >> > > > I think dwelling on the phrasing of non-English-native who works in >> a >> > > good >> > > > intention and raised real concerns about handling incoming code sets >> > > really >> > > > bad example and is discouraging to many who will read it - is this >> what >> > > > **we** want the Airflow Community to be? >> > > > >> > > > Best, >> > > > Michal >> > > > >> > > > >> > > > >> > > > czw., 8 paź 2026, 17:34 użytkownik Jarek Potiuk <[email protected]> >> > > napisał: >> > > > >> > > >> Ash, >> > > >> >> > > >> I think some clarifications would be beneficial. I did not find >> > anything >> > > >> rude in Vlada's message, it was polite and asking for help. I >> > > >> think responding with aggression is a bad idea especially that we >> > > wanted to >> > > >> cooperate with our stewards. >> > > >> >> > > >>> Also: You work for trillion dollar company. If this is important >> to >> > > >> >> > > >> Vlada does not work at Google. Vlada works for EPAM and Google >> hires >> > > EPAM >> > > >> to work on many things - like they used to pay Polidea and my team. >> > > About >> > > >> 50% of the provider code is Googles. >> > > >> >> > > >>> Google: show us the money. Give us the credits so every PR against >> > the >> > > >> Google PR can run against real GCP resources. >> > > >> >> > > >> This has been offered multiple times in the past; we were the ones >> > > unable >> > > >> to consume it. And yes as I understand it, the credits to run all >> that >> > > will >> > > >> happen and are part of this message. And what we miss is wiring >> > running >> > > the >> > > >> system tests (not only for Google but for other providers) in the >> ASF >> > > >> >> > > >> Also just to remind you Ash and everyone. Google is paying me >> **every >> > > >> single month** - similarly as Astronomer so that I can spend whole >> my >> > > time >> > > >> on Airflow (incluiding the 50% of extra time). They are also >> platinum >> > > >> sponsors of Airflow Summit. They are already showing the money. The >> > fact >> > > >> that you are not paying attention to those facts does not mean >> that it >> > > is >> > > >> not happening, Ash. Not mentioning the work on AIP-85. You should >> > > consider >> > > >> if you are not biased, or if you maybe should pay more attention. >> > > >> >> > > >> And just to be very clear: Vlada did exactly what we call stewards >> > for. >> > > She >> > > >> is taking care of issues she sees with maintaining the Google >> > provider. >> > > If >> > > >> you look at AIP-95 (Co-created by Vikram, Elad, yourself, Kaxil and >> > > me), it >> > > >> mentions a number of metrics (that we actually haven't checked), >> and >> > it >> > > >> focuses mostly on "Have PRs reviewed and merged" and "Issues >> handled". >> > > Not >> > > >> "Google team must merge them" - but "they are merged". And this >> > happens >> > > - >> > > >> partially with Google team support, partially mine, partiall >> Shahars - >> > > and >> > > >> few other people. >> > > >> >> > > >> If you think the Google-hired team isn't doing its job, ideally get >> > > those >> > > >> metrics and compare them against other stewards. >> > > >> >> > > >> I recommend basing such discussions on facts rather than personal >> > > feelings. >> > > >> And focus on solving issues not how somehow who is not English >> native >> > > puts >> > > >> they words into writing. Be empathetic and try to understand that >> > people >> > > >> might simply come from different cultures and communicate >> differently >> > - >> > > >> especially in writing. >> > > >> >> > > >> J. >> > > >> >> > > >> >> > > >> >> > > >> On Thu, Oct 8, 2026 at 5:08 PM Ash Berlin-Taylor <[email protected]> >> > > wrote: >> > > >> >> > > >>> Hi Ulada, >> > > >>> >> > > >>> You have really, really missed the mark in the tone you used on >> > **your >> > > >>> very first email** to the dev list. >> > > >>> >> > > >>> "We want a clear baseline” - who is “we"? Who are you even? I >> find >> > > this >> > > >>> very rude phrasing, and almost totally blind to how the Airflow >> > > community >> > > >>> works. Phrase it as “I would like to propose” and it reads much >> > better. >> > > >>> >> > > >>> Also: You work for trillion dollar company. If this is important >> to >> > > >>> Google: show us the money. Give us the credits to let every pr to >> the >> > > >>> Google PR run against real GCP things. >> > > >>> >> > > >>> Thanks, >> > > >>> Ash >> > > >>> >> > > >>>> On 5 Oct 2026, at 14:46, Ulada Zakharava (xWF) < >> > [email protected] >> > > > >> > > >>> wrote: >> > > >>>> >> > > >>>> Hi everyone, >> > > >>>> >> > > >>>> We’ve noticed an increasing number of PRs submitted to the Google >> > > >>> Provider >> > > >>>> where contributors—often using AI tooling—don't have access to a >> GCP >> > > >>>> project to test their changes. >> > > >>>> >> > > >>>> While we love community contributions, many of these PRs touch >> > > critical >> > > >>>> logic like authentication, base hooks, or core operator behavior >> > with >> > > >>> only >> > > >>>> mocked unit tests. Maintainers are regularly asked to pull these >> > > >> branches >> > > >>>> and run manual or system tests to see if they actually work >> against >> > > >> live >> > > >>>> APIs. >> > > >>>> >> > > >>>> Simply put, this doesn't scale. Pulling branches to run ad-hoc >> > system >> > > >>> tests >> > > >>>> takes significant time, and foundational changes often require >> > running >> > > >>>> multiple full suites. Merging without verification isn't an >> option >> > > >>> either, >> > > >>>> as catching regressions during release candidate testing blocks >> > > >> releases >> > > >>>> and creates painful reverts. >> > > >>>> Proposed Rule >> > > >>>> >> > > >>>> We want to agree on a clear baseline for PRs touching functional >> or >> > > >> auth >> > > >>>> logic in the Google Provider: >> > > >>>> >> > > >>>> 1. >> > > >>>> >> > > >>>> *Proof of testing required:* Authors must test against a real >> GCP >> > > >>>> project and include proof (logs, CLI output, or run results) in >> the >> > > >> PR >> > > >>>> description. >> > > >>>> 2. >> > > >>>> >> > > >>>> *Untested PRs will be put on hold:* Changes affecting runtime >> > > >> behavior >> > > >>>> without evidence of testing shouldn't be reviewed or merged >> until >> > > >>> validated. >> > > >>>> 3. >> > > >>>> >> > > >>>> *Keep system tests up to date:* New operators or behavior >> changes >> > > >>> should >> > > >>>> include or update automated system tests, or at minimum show >> manual >> > > >>>> end-to-end run results. >> > > >>>> >> > > >>>> What we're doing on our end >> > > >>>> >> > > >>>> We understand that not everyone has an active GCP environment. On >> > the >> > > >>>> Google/maintainer side, we’re looking into ways to make it >> easier to >> > > >>>> trigger specific existing system tests directly on PR branches >> via >> > CI >> > > >> for >> > > >>>> trusted contributors. Until that workflow is ready, the >> > responsibility >> > > >> of >> > > >>>> proving a change works has to sit with the author. >> > > >>>> >> > > >>>> What do you all think? We'd love to hear feedback before updating >> > the >> > > >>>> provider contribution docs. >> > > >>>> >> > > >>>> Best regards, >> > > >>>> >> > > >>>> Ulada >> > > >>> >> > > >>> >> > > >>> >> --------------------------------------------------------------------- >> > > >>> To unsubscribe, e-mail: [email protected] >> > > >>> For additional commands, e-mail: [email protected] >> > > >>> >> > > >>> >> > > >> >> > > >> > > >> > >> >
