On Wed, Nov 26, 2014 at 8:01 PM, Joaquin Oltra Hernandez <[email protected]> wrote: > TLDR: > > phantomjs cucumber tests ~20% faster that firefox (100% less annoying) > phantomjs cucumber tests have more failing tests than firefox > cucumber tests have tags. > > we should tag important/fast tests, and run them more often in dev > > I've been experimenting with the browser tests to see if it would make any > difference running them headlessly or w/ phantomjs in speed of execution. > > First, in OSX it is not supported running the cucumber tests in headless > mode (env var HEADLESS=true with any browser), so I haven't been able to > time that.
Try BROWSER=phantomjs I'm not sure if HEADLESS=true is needed any more.. > > When running the cucumber tests on my machine, my results have been 19m30s > for Firefox, and 15m for phantomjs. > > It is not a huge improvement, but still, around 23% speed up, so it is worth > it. Also, the browser window is not stealing focus from you while you work > every time a new test launches, so I would say that is a major win, at least > for me. > > I'll investigate a bit more since I get more failing tests in phantom js > than with firefox (28 vs 13) and also why I'm getting all those failing > tests even in firefox. If anybody has faced this issues, I'd love some help > (https://gist.github.com/joakin/853176ebf309798085f6) > > About executing an important subset, as we discussed, browser tests have > tags both for features and scenarios, and you can execute only tests with a > certain tag by doing bundle exec cucumber --tags @tagname for example. You > can see an example of tagging feature and scenario in mainmenu.feature for > example: > > @chrome @en.m.wikipedia.beta.wmflabs.org @firefox @test2.m.wikipedia.org > @vagrant > Feature: Menus open correct page for anonymous users > ... > @extension-geodata > Scenario: Nearby link in menu > ... > > I think it would benefit us a lot to tag both important and fast tests, so > that we run them more often in development and catch more regressions > earlier. I agree. @fast and @important tags would be useful. > > To run browser tests in mobilefrontend with phantomjs installed just do > BROWSER=phantomjs make cucumber > > More info: > > https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Running_tests#Running_the_tests I've cc'ed the qa and mobile-l public mailing lists as I think this is a useful discussion and will gather more expertise :). _______________________________________________ Mobile-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mobile-l
