Looping in QA list and dropping our team private list. <quote name="Jon Robson" date="2015-09-03" time="11:45:47 -0700"> > Dear Greg, and anyone else that is involved in deployment > > This is a follow-up from Dan Duvall's talk today during the metrics > meeting about voting browser tests. > > Background: > The reading web team this quarter with the help of Dan Duvall has made > huge strides in our QA infrastructure. The extensions Gather, > MobileFrontend and now the new extension QuickSurveys are all running > browser tests on a per commit basis. A selected set of MobileFrontend > @smoke tests (a selected subset of all he tests) are running in > 15minutes on every commit and the entire set of Gather browser tests > is running in around 21minutes. It marginally slows down getting > patches deployed... but I think this is a good thing. The results > speak for themselves. > > In the past month (August 4th-September 4th) only 3/33 builds failed > for MobileFrontend's daily smoke test build [1] (all 3 due to issues > with the Jenkins infrastructure). For the full set of tests only 10/33 > failed in the Chrome daily build [3], 8 of which were due to tests > being flakey and needing improvement or issues with the Jenkin > infrastructure and the two others serious bugs [4,5] brought about by > work the performance team had been doing that we were able to fix > shortly after. > > In Firefox [2] there were only 6 failures and only 2 of these were > serious bugs, again caused by things outside MobileFrontend [4,6]. One > of these was pretty serious - we had started loading JavaScript for > users with legacy browsers such as IE6. These were caught prior to the > daily builds when suddenly our MobileFrontend commits would not merge. > > The future!: > Given this success: > 1) I would like to see us run @integration tests on core, but I > understand given the number of bugs this might not be feasible so far. > 2) We should run @integration tests prior to deployments to the > cluster via the train and communicate out when we have failures (and > make a decision to push broken code) > 3) I'd like to see other extensions adopt browser test voting on their > extensions. Please feel free to reach out to me if you need help with > that. The more coverage across our extensions we have, the better. > > We really have no excuse going forward to push broken code out to our > users and at the very least we need to be visible to each other when > we are deploying broken code. We have a responsibility to our users. > > Thoughts? Reactions? Who's with me?! > > [1] > https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/ > [2] > https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/ > [3] > https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/ > [4] https://phabricator.wikimedia.org/T108045 > [5] https://phabricator.wikimedia.org/T108191 > [6] https://phabricator.wikimedia.org/T111233 > > _______________________________________________ > Mobile-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D | _______________________________________________ Mobile-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mobile-l
