When we began creating cross-browser regression tests using Selenium, we kept them all in a repository named '/qa/browsertests'. We created regression tests for features across all WMF development teams and Platform functions, from the appearance of Preferences to navigating UploadWizard to ArticleFeedback and PageTriage to MobileFrontend to UniversalLanguageSelector etc. etc. etc. We run these tests against some combinations of environments in beta labs cluster, test2wiki, and production.
There will probably always be a need for a grab-bag repo of general cross-browser regression tests, but managing tests in this way has some drawbacks: * builds run for all features at the same time, regardless of the development process for each feature * failed builds can be caused by any feature, and it takes some effort to find the source of any failed build We have taken some steps recently to improve how we manage and run these tests. In particular, we have begun moving certain browser tests out of the /qa/browsertests repo and into the repos of the particular extensions or features being tested. From there, we are running individual builds for those features in the appropriate test environments. Our first model for doing this was MobileFrontend. The browser tests for MobileFrontend have resided in /MobileFrontend/tests/acceptance for some time now, and building from that repo has proven quite effective. Not long ago Nik Everett created an extensive set of browser tests for CirrusSearch and very quickly put them under /CirrusSearch/tests/browser. Work on CirrusSearch is proceeding quickly and these tests fail often for now, so it is nice to have them in a repo where they can be examined outside of the main /qa/browsertest builds. Back in May of this year we created a number of browser tests for the UniversalLanguageSelector with help from Runa Bhattacharjee and Niklas Laxström, and Amir Aharoni has been doing work on those in the time since. Recently at the request of the Language team we moved the ULS browser tests to the repo at UniversalLanguageSelector/tests/browser, and we're running those in their own build as well. Finally, we have been creating some browser tests for VisualEditor that will be particularly useful for cross-browser acceptance testing and regression testing. Of course the tests are valuable right now as well, they help us identify important issues like https://bugzilla.wikimedia.org/show_bug.cgi?id=53360. These tests today reside under VisualEditor/modules/ve-mw/test/browser. Rachel Thomas, our intern with the Outreach Program for Women has contributed a number of VisualEditor browser tests. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
