Small poking at important things from holidays while having a coffee at a place with internet :)
+1 on having both. I have no strong opinion on the language and Playwright. But there are few things for consideration: We should have many, many more unit tests than e2e tests. This is the classic test pyramid case: The Practical Test Pyramid https://share.google/YHVIt8EBptT1yYFur The e2e tests are slow, brittle and difficult to maintain and we should have rather small number of those - mostly green path on all major functionalities. Possibly with some cross browser checks - but those are very difficult to get right. It will be a pain to maintain those. Like our k8s e2e tests - when they break (rarely) people will have hard time to create and run the tests so the tooling around that has to drive people who will be reproducing the failures by hand - how to set it up and how to reproduce it and how to iterate as fast as possible. Keeping the tests stable and side-effect free will be a challenge - so likely a good 'reset the environment' before each test is a must. Also they need to be run with all UI changes in CI which will make UI PR run longer. It's an imperative to get them run not only in canary builds but in all PRs for ui and API and likely whenever any core model change. It will slow down UI development and create frustration for contributors unfamiliar with the test framework so we need to have at least a few people proficient in intricacies of the test framework dedicated and willing to help those who will struggle with issues. Language is secondary - e2e tests will require all stack - Python (backend) and Typescript and react experiences anyway. Most likely people who write UI should be involved so Typescript is fine, and also even those who do not know TypeScript will be able to fix some obvious cases. J. śr., 24 wrz 2025, 09:50 użytkownik Shubham Raj <[email protected]> napisał: > This addition will be really nice. > Also, at this point of 3.x, I think UI has been relatively stable and > writing e2e would be a good starting point. Unit tests can go parallel. > It's right that people maintaining CI/CD would be first persons to see any > breakage, but I feel writing in ts will be easier for folks working on > frontend, and also for the new contributors who would like to contribute in > this UI e2e tests coming from cypress or similar experience, since majority > of other choices do come with ts/js support. > I would be happy to help with the e2e tests. We should probably go in > phases with P0/1/2. > > Thanks. > > > On Wed, Sep 24, 2025 at 9:01 PM Pierre Jeambrun <[email protected]> > wrote: > > > I also think that having a good unit test coverage + e2e tests would > really > > help maintaining quality of releases and prevent regressions early, > making > > all that much more manageable. > > > > My two cents regarding the questions above: > > > > - > > Should we prioritize expanding unit test coverage first, or work on > unit > > and E2E tests in parallel? > > > > Both in parallel sounds good to me. They do not depend on each other > (At > > least in my mind) > > - > > For E2E testing, is Playwright a good choice, or are there other > > preferred tools we should explore? > > > > I like playwright and I feel this is the modern solution to go to. > > - > > If Playwright, should we use TypeScript or Python as language for > tests? > > > > Good arguments mentioned above for both sides. I personally think > (maybe > > wrongly), that ideally people maintaining / developing e2e tests will be > > people with > > front-end knowledge. (events, identifiers, selectors, css styling, > > browser local states, etc..), so I tend to think that those people will > > most likely come from the front-end community and > > having that in typescript would probably help them get started. > > > > > > On Wed, Sep 24, 2025 at 10:01 AM Amogh Desai <[email protected]> > > wrote: > > > > > Thanks for starting this discussion, Rahul. > > > > > > I am aligned with your thoughts, we have seen many many UI issues > lately > > > which I believe we should prioritize fixing or at least reducing going > > > forward. > > > > > > To answer your question(s) -- > > > > > > * I think the unit tests and e2e / user flow tests as orthogonal, so > they > > > can be > > > developed in parallel. > > > > > > * While I do not have real experience with Playwright but from what I > > hear > > > / read on reddit, > > > github, and related forums, it looks like Playwright can do similar job > > as > > > alternatives such as > > > Selenium, Cypress, while being more beginner friendly, which could play > > > better in our favour. > > > > > > * Again, not very strong on this but I feel that having Typescript as > the > > > language of choice would limit > > > the number of contributors we get to help on this. Python would be my > > > weapon of choice, given > > > the "testing" is not limited by it and it potentially could encourage > > > broader participation. > > > > > > (Folks active on CI/CD(atleast me), are not very comfortable with > > > Typescript tests compared to > > > Python) > > > > > > One additional point: It would be valuable if we extract the e2e / user > > > flow testing into a GH job that > > > a release manager could run as "sort" of last validation before > releasing > > > to avoid any last > > > minute issues. > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > > > > On Tue, Sep 23, 2025 at 9:15 PM Brent Bovenzi via dev < > > > [email protected]> wrote: > > > > > > > I agree that we need to improve UI tests. Happy to make that a > priority > > > > during 3.1.x and 3.2 development. > > > > > > > > Personally, I will start with expanding unit tests first. But that > > > doesn't > > > > mean others can't get started on E2E tests. > > > > > > > > I lean towards using typescript for playwright. With E2E we are > testing > > > the > > > > UI more than we are testing the API so I think the code should sit > > closer > > > > to the UI code. > > > > > > > > On Tue, Sep 23, 2025 at 9:38 AM Vincent Beck <[email protected]> > > > wrote: > > > > > > > > > Not answering any of these questions (sorry) but strong +1 on > adding > > > > > end-to-end (E2E) tests. This will significantly make our UI more > > robust > > > > but > > > > > also will speed up some of our work. An example is front-end > > dependency > > > > > upgrades by dependabot. Today we have no way of knowing whether > these > > > PRs > > > > > break the UI, in order to merge these PRs we need to compile the > > > > front-end > > > > > assets and test manually whether the UI work. Having end-to-end > tests > > > on > > > > > the front-end and making them run in the CI would speed up this > very > > > > > example (on top of increasing code robustness, increase our > > confidence > > > > etc). > > > > > > > > > > On 2025/09/23 11:03:40 Rahul Vats wrote: > > > > > > Hi all, > > > > > > > > > > > > With Airflow 3.0, the UI has undergone a complete revamp, and new > > > > changes > > > > > > are naturally more prone to regressions. We've already seen more > UI > > > > > > regressions that are hard to catch with our current limited test > > > > > coverage. > > > > > > Manual testing is also challenging given the wide variety of > > > workflows > > > > > and > > > > > > screens in the UI. > > > > > > > > > > > > Currently, the Airflow UI has a few unit tests using Vitest and > > React > > > > > > Testing Library, but coverage is quite limited — only about 4 > test > > > > files > > > > > > covering major pages like DagsList and TaskInstance. Many > critical > > > > > > pages (Connections, > > > > > > Variables, Pools, Dashboard, etc.) have no test coverage at all. > > > > > > > > > > > > I’m exploring a two-pronged approach to improve UI testing: > > > > > > > > > > > > 1. > > > > > > > > > > > > Expand unit test coverage for individual components and pages > > > > > > > > > > > > 2. > > > > > > > > > > > > Add end-to-end (E2E) tests using Playwright for complete user > > > > > workflows > > > > > > > > > > > > > > > > > > This would give us both component-level safety and full > > user-journey > > > > > > testing across browsers, helping catch regressions early. > > > > > > > > > > > > A few questions: > > > > > > > > > > > > - > > > > > > > > > > > > Should we prioritize expanding unit test coverage first, or > work > > > on > > > > > unit > > > > > > and E2E tests in parallel? > > > > > > > > > > > > - > > > > > > > > > > > > For E2E testing, is Playwright a good choice, or are there > other > > > > > > preferred tools we should explore? > > > > > > > > > > > > - > > > > > > > > > > > > If Playwright, should we use TypeScript or Python as language > > for > > > > > tests? > > > > > > > > > > > > - > > > > > > > > > > > > Any other suggestions to strengthen our UI testing strategy? > > > > > > > > > > > > > > > > > > I’d love to hear your thoughts on whether this would be valuable > > for > > > > the > > > > > > project before diving into implementation details. > > > > > > > > > > > > Regards, > > > > > > Rahul Vats > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [email protected] > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > >
