It really saddens me how very few engineers seem to care about browser tests. Our browser tests are failing all over the place. I just saw this bug [1] which has been sitting around for ages and denying us green tests in Echo one of our most important features.
How can we change this anti-pattern? Dan Duval, would it make sense to do a survey as you did with Vagrant to understand how our developers think of these? Such as who owns them... who is responsible for a test failing... who writes them... who doesn't understand them.. why they don't understand them etc...? [1] https://phabricator.wikimedia.org/T72298 On Thu, Mar 26, 2015 at 5:54 PM, Dan Duvall <[email protected]> wrote: > On Thu, Mar 26, 2015 at 3:38 PM, Jon Robson <[email protected]> wrote: > > The lack of replies from any mobile team members seems to suggest I'm the > > only one watching this. In that case I'd rather we split up the big > > MobileFrontend job into various jobs based on certain features. How can > we > > get more people caring about browser tests? Naming and shaming? > > We have a couple projects in the works that will hopefully help.[1][2] > They don't go so far as to name and shame (we'll leave that up to you > :), but they should promote more ownership over browser tests, better > communication of failure, and analysis of failure over time and by > test function (feature vs smoke vs regression). > > If many of these browser tests are serving as regression/smoke tests, > it might be worthwhile to not only separate them out, but to remove > some of the layers of abstraction to make tests more maintainable. I > often try to think about tests in terms of a value-to-debt ratio (i.e. > "How likely is it that I'll have to fix or refactor this test down the > road and is it worth it?") and, while I find quite a bit of value in > Cucumber for the sake of acceptance(-esque) testing (it keeps me > mindful of the UX),[3] it introduces quite a bit of debt in its layers > of abstraction and indirection (it's always a red flag when you have > to grep to find your test implementation). Even its creators believe > the value of Cucumber lies in its collaboration/planning capacities, > not in its function as a test runner.[4] > > It's entirely possible, as of the 1.0.0 release, to use MW-Selenium > without Cucumber, so perhaps we could experiment with simple RSpec > examples for these types of tests. > > Tooling aside, there are broader discussions that need to take place > among those that are practicing or have practiced TDD/BDD in the > organization and how we might (or might not) promote theses practices > among others. I'll be trying to form such a group next quarter (will > it be a 'guild'?, a 'league'?, no need for tough decisions just > yet[5]), so let's maintain a dialogue on that front if you're > interested and willing. > > [1] https://phabricator.wikimedia.org/T88706 > [2] https://phabricator.wikimedia.org/T89375 > [3] > https://gerrit.wikimedia.org/r/#/c/196492/10/features/role_settings.feature > [4] > https://cukes.info/blog/2014/03/03/the-worlds-most-misunderstood-collaboration-tool > [5] https://www.youtube.com/watch?v=6KSiyaqnZYs > > > Practically, the current big job [1] we have has far too many false > > positives and it is just noise to me. I was paying attention to the smoke > > tests [2] but I was told that was an upstream bug and haven't been > watching > > it since. Any idea why that has been failing recently? Nothing has broken > > here... > > > > [1] > > > https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/ > > [2] > > > https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/ > > > > Have you tried to reproduce these failures by executing the tests > locally, against either Beta or your local MW? I'll be in the office > tomorrow if you want advice on how to best go about it. > > Dan > > -- > Dan Duvall > Automation Engineer > Wikimedia Foundation > -- Jon Robson * http://jonrobson.me.uk * https://www.facebook.com/jonrobson * @rakugojon
_______________________________________________ Mobile-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mobile-l
