Michael added a comment.
TL;DR: We should take some time to consider alternatives to
Selenium+Webdriver.IO before rewriting a large number of Ruby tests.
Reevaluate our choice of browser testing framework
--------------------------------------------------
We are about to commit a significant amount of time into rewriting //a lot//
of ruby tests. Before we commit to this massive marathon, we should ask
ourselves if the intended goal format, which we have been using for the last 4
years, is still our best option, given our experiences with it so far and how
the alternatives developed.
Since the beginning of this year, there were some investigations into Cypress
by several WMDE developers, with a generally favorable impression. Also, the
Query Builder repository uses Cypress as its browser testing framework.
So if ever consider switching away from Selenium+Webdriver.io, then it should
be //before// we rewrite a large number of tests. Meaning: //Now//.
Problems with the current setup - reasons to reconsider
-------------------------------------------------------
1. **flakiness**: The current tests are sometimes quite flaky, causing
unreleated test failures in Jenkins.
- There is a good chance that part of this not due to Webdriver, but the CI
VM doing strange things: T281122 <https://phabricator.wikimedia.org/T281122>
- OTOH, Lexeme daily tests on beta currently forget that they just clicked
the login button and then fail because they can't click the login button
<https://phabricator.wikimedia.org/T277862>
2. **deprecation of a central component**: We may have to rewrite all
Selenium/Webdriver Browser tests to use `async/await` everywhere at some point:
> Caution:
> As of 14/04/2021 sync mode will not be supported anymore starting from
Node.js v16 due to changes in Chromium. If you start a project with WebdriverIO
we recommend to not use sync mode anymore. The community is working on a
transition plan, read more on that in this RFC
<https://github.com/webdriverio/webdriverio/discussions/6702>.
- Sync vs. Async Mode <https://webdriver.io/docs/sync-vs-async/>
Though maybe there will be a codemod to mass-rewrite all tests. And given the
speed at which WMF updates their CI infrastructure, it might be some time till
we hit Nodejs 16, so this is probably not an issue for a while. That being
said, the new `async/await` style of writing these tests is //really//
unappealing, because we need to add `await` to basically //everything//. How
this would look can be seen in a Wikidata Bridge patch
<https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/565091/>.
3. **DX**: Finally, it seems that the Dev Experience writing browser tests is
currently generally low, and it is considered a rather disliked chore.
Improving this aspect could improve overall dev satisfaction and help
proliferate better development practices such as Outside-In TDD.
Alternatives to Consider
------------------------
The two main alternatives to be consider would be Cypress and Puppeteer (and
maybe Playwright).
- Cypress is already used by the Query Builder project on the WMF CI
infrastructure
- Puppeteer is already used by the WMDE FUN team: "dandy" test runner
<https://gitlab.com/fun-tech/dandy>
- (AFAICT, WMDE has no prior experience with Playwright)
WD/WB experience with Cypress
-----------------------------
- Cypress was evaluated during a "Ginger Beer and Cake" event in early 2021
<https://docs.google.com/document/d/1g9vpW_WOsVugDL5tq9v_swcNa8At6F_Jpc6Z21QndXg/edit>
- It is used in the Query Builder repository within the WMF CI infrastructure
and works with just the standard config
<https://github.com/wikimedia/integration-config/blob/9ea60e7f33cdac0bf680e1ca20116dc1e6db0c91/zuul/layout.yaml#L8976>
Some Cypress Advantages we noticed
----------------------------------
- interactive test runner/dashboard is really nice in terms of DX
- request mocking can be very useful for a frontend project (Though
webdriver.io seems to have acquired some capabilities in this regard in one of
the v6 minor releases)
- works out-of-the-box with existing WMF CI infrastructure and also locally
- there is some specialized work happening towards enabling **first-class vue
component testing**, based on this VueConfUS 2020 talk
<https://www.youtube.com/watch?v=Abwi_X107GY>
Previous work by WMF:
---------------------
- Selenium WebdriverIO was chosen as a Ruby alternative in April 2017
<https://lists.wikimedia.org/hyperkitty/list/[email protected]/message/WAEY3SBXD3ZJHJL4Q3OUJWA65JYRQ72F/>
- Puppeteer and Playwright didn't exist back then, Cypress did but
apparently wasn't evaluated? <https://filipin.eu/selenium-javascript>
- In Summer 2020 WMF had Google Summer of Code students evaluate Cypress,
Puppeteer and Playwright as alternatives to WebdriverIO: Evaluate WebdriverIO
replacements for our browser automation framework
<https://phabricator.wikimedia.org/T247835>
- Playwright Report
<https://github.com/hariclerry/mediawiki/blob/master/tests/playwright/Report.md#playwright-vs-webdriverIO>
- Puppeteer Report <https://www.sohamp.dev/blog/2020-08-24-wmf-gsoc2020/>
- They seemed to have been unable to get Cypress to work. But when I
updated Cypress and resubmitted their patch, it work perfectly fine out of the
box: Core Cypress test in CI
<https://gerrit.wikimedia.org/r/c/mediawiki/core/+/683416>
Next steps
----------
- We need to decide if we //want// to spend some time evaluating the
alternatives
- We should, because of the reasons mentioned above
- We should write a short ADR on either why we want to continue using
WebdriverIO + Selenium, or why decided to go with one of the alternatives
TASK DETAIL
https://phabricator.wikimedia.org/T180228
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Michael
Cc: Tonina_Zhelyazkova_WMDE, Jdforrester-WMF, Lydia_Pintscher, Addshore,
Lucas_Werkmeister_WMDE, alaa_wmde, WMDE-leszek, Michael, Ladsgroup, gerritbot,
Aklapper, zeljkofilipin, Invadibot, maantietaja, Akuckartz, Nandana, Lahi,
Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS,
Jonas, Wikidata-bugs, aude, Mbch331
_______________________________________________
Wikidata-bugs mailing list -- [email protected]
To unsubscribe send an email to [email protected]
