Hi Everyone, Thank you all for the fantastic feedback and strong support! The community consensus is clear and encouraging:
- High priority for improving UI testing - Playwright + TypeScript for e2e testing - Parallel development of unit and e2e tests - Test pyramid approach with focused e2e scope - Multiple volunteers ready to contribute! *Addressing Key Concerns* Important points on e2e complexity are well taken. I completely agree that we should follow the test pyramid. - E2E tests should cover only critical workflows (~10-15 scenarios) - Unit tests are the foundation and should be the primary focus and we should have an enforcement so that every UI change comes with a unit test. - CI integration needs careful planning to avoid slowing development. *Next Steps* - I plan to work on a POC for E2E tests and will raise a PR for review soon. After that, I'll share a detailed implementation plan and coordinate with volunteers who offered to help. - In the meantime, I suggest we should continue expanding unit test coverage as planned. It was great meeting you all at the Airflow Summit. Momentum here is fantastic, and I'm excited to work with everyone who offered to help make Airflow 3 UI much more reliable! Regards, Rahul Vats On Thu, 25 Sept 2025 at 07:49, Vikram Koka via dev <dev@airflow.apache.org> wrote: > Rahul, > Thank you for starting this thread. > > Strong +1 from me as well. > I not only agree with this, but I believe this needs to be a high priority > going forward and before 3.2. > > I also agree with the comments on the "test pyramid" approach, of there > needing to be FAR more unit tests as compared to end to end tests. > I also believe that a certain level of rigor around this i.e. some form of > enforcement to ensure that all new UI changes come with a unit test. > > I would also add the Core API changes (for UI) to the mix here, since a > fair number of UI changes requires underlying API changes. > > Vikram > > > On Thu, Sep 25, 2025 at 12:04 AM Amogh Desai <amoghde...@apache.org> > wrote: > > > Valid points. > > > > I was just expressing concern from a CI/CD failure perspective but in no > > way > > was I objecting to usage of typescript. > > > > Anything that works for the team that owns most of the tests works by me! > > > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Thu, Sep 25, 2025 at 1:39 AM Jens Scheffler <jsche...@apache.org> > > wrote: > > > > > Echoing a full agreement to all statement before. > > > > > > I also would emphasize the test pyramid. As we really have a low > > > coverage we should maybe also have a metric. I would also propose that > > > we focus more on testing coverage == hopefully also improved quality > > > preventing regression in 3.2. One option could a bit of force to force > > > all new changes applied to UI to request a unit test being added (all > > > changes TS/x files need to have a corresponding .test.ts/x file) except > > > for cases where it does not make sense we allow exclusion of the rule > by > > > adding a label by maintainers. (just want to drop the idea, might be > > > many will hate it) > > > > > > For language also have no strong preference but even though I am a TS > > > noob I see that testing UI is closer to UI development, to using Python > > > for TS UI testing might make a knowledge gap also for potential utility > > > code. So as most voices would propose TS (maybe then finally I get to > > > understand it) > > > > > > Jens > > > > > > On 24.09.25 18:33, Jarek Potiuk wrote: > > > > Small poking at important things from holidays while having a coffee > > at a > > > > place with internet :) > > > > > > > > +1 on having both. I have no strong opinion on the language and > > > Playwright. > > > > > > > > But there are few things for consideration: > > > > > > > > We should have many, many more unit tests than e2e tests. This is the > > > > classic test pyramid case: The Practical Test Pyramid > > > > https://share.google/YHVIt8EBptT1yYFur > > > > > > > > The e2e tests are slow, brittle and difficult to maintain and we > should > > > > have rather small number of those - mostly green path on all major > > > > functionalities. Possibly with some cross browser checks - but those > > are > > > > very difficult to get right. > > > > > > > > It will be a pain to maintain those. Like our k8s e2e tests - when > they > > > > break (rarely) people will have hard time to create and run the tests > > so > > > > the tooling around that has to drive people who will be reproducing > > the > > > > failures by hand - how to set it up and how to reproduce it and how > to > > > > iterate as fast as possible. > > > > > > > > Keeping the tests stable and side-effect free will be a challenge - > so > > > > likely a good 'reset the environment' before each test is a must. > Also > > > they > > > > need to be run with all UI changes in CI which will make UI PR run > > > longer. > > > > It's an imperative to get them run not only in canary builds but in > all > > > PRs > > > > for ui and API and likely whenever any core model change. It will > slow > > > down > > > > UI development and create frustration for contributors unfamiliar > with > > > the > > > > test framework so we need to have at least a few people proficient in > > > > intricacies of the test framework dedicated and willing to help those > > who > > > > will struggle with issues. > > > > > > > > Language is secondary - e2e tests will require all stack - Python > > > (backend) > > > > and Typescript and react experiences anyway. Most likely people who > > write > > > > UI should be involved so Typescript is fine, and also even those who > do > > > not > > > > know TypeScript will be able to fix some obvious cases. > > > > > > > > J. > > > > > > > > śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj < > shubham...@apache.org> > > > > napisał: > > > > > > > >> This addition will be really nice. > > > >> Also, at this point of 3.x, I think UI has been relatively stable > and > > > >> writing e2e would be a good starting point. Unit tests can go > > parallel. > > > >> It's right that people maintaining CI/CD would be first persons to > see > > > any > > > >> breakage, but I feel writing in ts will be easier for folks working > on > > > >> frontend, and also for the new contributors who would like to > > > contribute in > > > >> this UI e2e tests coming from cypress or similar experience, since > > > majority > > > >> of other choices do come with ts/js support. > > > >> I would be happy to help with the e2e tests. We should probably go > in > > > >> phases with P0/1/2. > > > >> > > > >> Thanks. > > > >> > > > >> > > > >> On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun < > > pierrejb...@gmail.com> > > > >> wrote: > > > >> > > > >>> I also think that having a good unit test coverage + e2e tests > would > > > >> really > > > >>> help maintaining quality of releases and prevent regressions early, > > > >> making > > > >>> all that much more manageable. > > > >>> > > > >>> My two cents regarding the questions above: > > > >>> > > > >>> - > > > >>> Should we prioritize expanding unit test coverage first, or > work > > on > > > >> unit > > > >>> and E2E tests in parallel? > > > >>> > > > >>> Both in parallel sounds good to me. They do not depend on each > > > other > > > >> (At > > > >>> least in my mind) > > > >>> - > > > >>> For E2E testing, is Playwright a good choice, or are there > other > > > >>> preferred tools we should explore? > > > >>> > > > >>> I like playwright and I feel this is the modern solution to go > > to. > > > >>> - > > > >>> If Playwright, should we use TypeScript or Python as language > for > > > >> tests? > > > >>> Good arguments mentioned above for both sides. I personally > think > > > >> (maybe > > > >>> wrongly), that ideally people maintaining / developing e2e tests > will > > > be > > > >>> people with > > > >>> front-end knowledge. (events, identifiers, selectors, css > > styling, > > > >>> browser local states, etc..), so I tend to think that those people > > > will > > > >>> most likely come from the front-end community and > > > >>> having that in typescript would probably help them get started. > > > >>> > > > >>> > > > >>> On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai < > amoghde...@apache.org> > > > >>> wrote: > > > >>> > > > >>>> Thanks for starting this discussion, Rahul. > > > >>>> > > > >>>> I am aligned with your thoughts, we have seen many many UI issues > > > >> lately > > > >>>> which I believe we should prioritize fixing or at least reducing > > going > > > >>>> forward. > > > >>>> > > > >>>> To answer your question(s) -- > > > >>>> > > > >>>> * I think the unit tests and e2e / user flow tests as orthogonal, > so > > > >> they > > > >>>> can be > > > >>>> developed in parallel. > > > >>>> > > > >>>> * While I do not have real experience with Playwright but from > what > > I > > > >>> hear > > > >>>> / read on reddit, > > > >>>> github, and related forums, it looks like Playwright can do > similar > > > job > > > >>> as > > > >>>> alternatives such as > > > >>>> Selenium, Cypress, while being more beginner friendly, which could > > > play > > > >>>> better in our favour. > > > >>>> > > > >>>> * Again, not very strong on this but I feel that having Typescript > > as > > > >> the > > > >>>> language of choice would limit > > > >>>> the number of contributors we get to help on this. Python would be > > my > > > >>>> weapon of choice, given > > > >>>> the "testing" is not limited by it and it potentially could > > encourage > > > >>>> broader participation. > > > >>>> > > > >>>> (Folks active on CI/CD(atleast me), are not very comfortable with > > > >>>> Typescript tests compared to > > > >>>> Python) > > > >>>> > > > >>>> One additional point: It would be valuable if we extract the e2e / > > > user > > > >>>> flow testing into a GH job that > > > >>>> a release manager could run as "sort" of last validation before > > > >> releasing > > > >>>> to avoid any last > > > >>>> minute issues. > > > >>>> > > > >>>> Thanks & Regards, > > > >>>> Amogh Desai > > > >>>> > > > >>>> > > > >>>> On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev < > > > >>>> dev@airflow.apache.org> wrote: > > > >>>> > > > >>>>> I agree that we need to improve UI tests. Happy to make that a > > > >> priority > > > >>>>> during 3.1.x and 3.2 development. > > > >>>>> > > > >>>>> Personally, I will start with expanding unit tests first. But > that > > > >>>> doesn't > > > >>>>> mean others can't get started on E2E tests. > > > >>>>> > > > >>>>> I lean towards using typescript for playwright. With E2E we are > > > >> testing > > > >>>> the > > > >>>>> UI more than we are testing the API so I think the code should > sit > > > >>> closer > > > >>>>> to the UI code. > > > >>>>> > > > >>>>> On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck < > vincb...@apache.org> > > > >>>> wrote: > > > >>>>>> Not answering any of these questions (sorry) but strong +1 on > > > >> adding > > > >>>>>> end-to-end (E2E) tests. This will significantly make our UI more > > > >>> robust > > > >>>>> but > > > >>>>>> also will speed up some of our work. An example is front-end > > > >>> dependency > > > >>>>>> upgrades by dependabot. Today we have no way of knowing whether > > > >> these > > > >>>> PRs > > > >>>>>> break the UI, in order to merge these PRs we need to compile the > > > >>>>> front-end > > > >>>>>> assets and test manually whether the UI work. Having end-to-end > > > >> tests > > > >>>> on > > > >>>>>> the front-end and making them run in the CI would speed up this > > > >> very > > > >>>>>> example (on top of increasing code robustness, increase our > > > >>> confidence > > > >>>>> etc). > > > >>>>>> On 2025/09/23 11:03:40 Rahul Vats wrote: > > > >>>>>>> Hi all, > > > >>>>>>> > > > >>>>>>> With Airflow 3.0, the UI has undergone a complete revamp, and > new > > > >>>>> changes > > > >>>>>>> are naturally more prone to regressions. We've already seen > more > > > >> UI > > > >>>>>>> regressions that are hard to catch with our current limited > test > > > >>>>>> coverage. > > > >>>>>>> Manual testing is also challenging given the wide variety of > > > >>>> workflows > > > >>>>>> and > > > >>>>>>> screens in the UI. > > > >>>>>>> > > > >>>>>>> Currently, the Airflow UI has a few unit tests using Vitest and > > > >>> React > > > >>>>>>> Testing Library, but coverage is quite limited — only about 4 > > > >> test > > > >>>>> files > > > >>>>>>> covering major pages like DagsList and TaskInstance. Many > > > >> critical > > > >>>>>>> pages (Connections, > > > >>>>>>> Variables, Pools, Dashboard, etc.) have no test coverage at > all. > > > >>>>>>> > > > >>>>>>> I’m exploring a two-pronged approach to improve UI testing: > > > >>>>>>> > > > >>>>>>> 1. > > > >>>>>>> > > > >>>>>>> Expand unit test coverage for individual components and > pages > > > >>>>>>> > > > >>>>>>> 2. > > > >>>>>>> > > > >>>>>>> Add end-to-end (E2E) tests using Playwright for complete > user > > > >>>>>> workflows > > > >>>>>>> > > > >>>>>>> This would give us both component-level safety and full > > > >>> user-journey > > > >>>>>>> testing across browsers, helping catch regressions early. > > > >>>>>>> > > > >>>>>>> A few questions: > > > >>>>>>> > > > >>>>>>> - > > > >>>>>>> > > > >>>>>>> Should we prioritize expanding unit test coverage first, or > > > >> work > > > >>>> on > > > >>>>>> unit > > > >>>>>>> and E2E tests in parallel? > > > >>>>>>> > > > >>>>>>> - > > > >>>>>>> > > > >>>>>>> For E2E testing, is Playwright a good choice, or are there > > > >> other > > > >>>>>>> preferred tools we should explore? > > > >>>>>>> > > > >>>>>>> - > > > >>>>>>> > > > >>>>>>> If Playwright, should we use TypeScript or Python as > language > > > >>> for > > > >>>>>> tests? > > > >>>>>>> - > > > >>>>>>> > > > >>>>>>> Any other suggestions to strengthen our UI testing > strategy? > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> I’d love to hear your thoughts on whether this would be > valuable > > > >>> for > > > >>>>> the > > > >>>>>>> project before diving into implementation details. > > > >>>>>>> > > > >>>>>>> Regards, > > > >>>>>>> Rahul Vats > > > >>>>>>> > > > >>>>>> > > > >> > --------------------------------------------------------------------- > > > >>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > >>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > > >>>>>> > > > >>>>>> > > > >> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > >
