So far I'm seeing some extremely positive results. The tests are running super fast (in phantomjs the smoke tests are taking less than 3 minutes...).
I've been manually running him whilst I do code review in parallel and he's already generated some interesting conversation on this patchset: https://gerrit.wikimedia.org/r/#/c/219249/ If you want to pair and get this to be less hacky I'm more than happy! On Thu, Jun 18, 2015 at 5:41 PM, Dan Duvall <[email protected]> wrote: > Nice work, Jon. > > I've opened a task for defining a JJB builder/template and getting something > like this into CI sooner rather than later.[1] I think your setup proves > that a set of well-groomed MW-Selenium integration tests can be stable > enough for this purpose, and we can start with an even smaller subset of > core tests for a pre-merge build. Of course this isn't something that we > planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so > 'soon'—but we can start with an experiment on Gather or MobileFrontend > tests, since their health has greatly improved, and see how it goes. > > [1] https://phabricator.wikimedia.org/T103039 > > On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson <[email protected]> wrote: >> >> So the script that actually runs the browser test is not in a generic >> useful form but it is: >> https://gist.github.com/jdlrobson/32b607f8009e897ee80c >> >> It uses the GerritCommandLine tool to do grabbing and reviewing >> https://github.com/jdlrobson/GerritCommandLine >> >> Ideally if we can use labs-tools-gerrit-to-redis for identifying >> patches and then pulling them down we wouldn't need the >> GerritCommandLine tool since the code to submit a review is pretty >> trivial and captured in this function: >> https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277 >> >> I've also put this in the task >> https://phabricator.wikimedia.org/T101069#1379462 >> >> We'll probably want an instance per extension, to simplify having to >> worry about dependencies (we can just run a git update on all >> extensions after each checkout) >> >> On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez >> <[email protected]> wrote: >> > Awesome Jon! I'm so happy to finally see this developing :DD >> > >> > Loving the : `Browserbot happy!` >> > >> > I've noticed it can report either the name of the failing test or the >> > full >> > log. What do you think if we show that, and a url with the pasted log >> > somewhere publicly to not put too much noise on the comments but still >> > be >> > able to see it? Something like >> > https://phabricator.wikimedia.org/paste/... >> > >> > +1 to where is the source. >> > +1 to documenting how you've set it all up on wiki somewhere. >> > >> > I also think we need a catchy phrase for the -1s! >> > >> > Thanks for you work on this, we'll get more focused time for it soon. >> > >> > On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith <[email protected]> >> > wrote: >> >> >> >> I agree with Florian everything that you've written should be in a >> >> public >> >> version control system. >> >> >> >> Second, I'd ask that you document your experiences so far in getting >> >> this >> >> set up and how it works so that other members of the vertical can help >> >> to >> >> maintain it moving forward. >> >> >> >> Third, great work!!1 >> >> >> >> <3 >> >> >> >> –Sam >> >> >> >> On Thu, Jun 18, 2015 at 7:09 AM, [email protected] >> >> <[email protected]> wrote: >> >>> >> >>> >> >>> > It's currently working via a script that you can find here: >> >>> > /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh >> >>> >> >>> It would be great to have the script in a public version control >> >>> system >> >>> (e.g. github?), especially for people, e.g. volunteers, who can't ssh >> >>> to >> >>> gather-browser-tests.eqiad.wmflabs[1] >> >>> >> >>> [1] all people, who're not members of >> >>> https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests >> >>> >> >>> Best, >> >>> Florian >> >>> >> >>> -----Original-Nachricht----- >> >>> Betreff: [WikimediaMobile] [Update] Browser tests per patch >> >>> Datum: Thu, 18 Jun 2015 03:27:32 +0200 >> >>> Von: Jon Robson <[email protected]> >> >>> An: "QA (software quality assurance) for Wikimedia projects." >> >>> <[email protected]>, mobile-l <[email protected]> >> >>> >> >>> Background: mobile wants to gain more confidence in its browser tests >> >>> by running a subset of browser tests on a case by case basis [0]. >> >>> >> >>> Good news: I've got a proof of concept running and Barry the browser >> >>> test bot has given some legitimate helpful reviews to Gather [1]. >> >>> >> >>> Even better news: It's proving itself valuable already [2]. >> >>> As you can see in the messages the bot has posted on [3] we have a >> >>> couple of options on display option format for his reviews. >> >>> >> >>> So.. hopefully this short experience has sold you all already. >> >>> >> >>> This script is currently a manual job and needs a bit of tweaking >> >>> before we can put it in a cron job/run it always - it needs to watch >> >>> for new commits and then run a modification of the above script on a >> >>> per case basis (if two versions of it run in parallel we have an >> >>> issue). >> >>> >> >>> Definitely something we should push for next sprint! >> >>> >> >>> Long live Barry bot! >> >>> >> >>> Devs... (everyone else now of what follows is likely to be useful): >> >>> I got the labs instance up and running on: >> >>> http://gather-browser-tests.wmflabs.org/wiki/Main_Page >> >>> >> >>> Most of you in readership team should be able to ssh >> >>> gather-browser-tests.eqiad.wmflabs >> >>> Let me know if you have no access. >> >>> >> >>> It's currently working via a script that you can find here: >> >>> /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh >> >>> >> >>> [0] https://phabricator.wikimedia.org/T100293 >> >>> [1] >> >>> >> >>> https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.com+status:open,n,z >> >>> [2] https://gerrit.wikimedia.org/r/#/c/218731/ >> >>> >> >>> _______________________________________________ >> >>> Mobile-l mailing list >> >>> [email protected] >> >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> Mobile-l mailing list >> >>> [email protected] >> >>> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >> >> >> >> >> >> >> _______________________________________________ >> >> Mobile-l mailing list >> >> [email protected] >> >> https://lists.wikimedia.org/mailman/listinfo/mobile-l >> >> >> > >> > >> > _______________________________________________ >> > Mobile-l mailing list >> > [email protected] >> > https://lists.wikimedia.org/mailman/listinfo/mobile-l >> > >> >> >> >> -- >> Jon Robson >> * http://jonrobson.me.uk >> * https://www.facebook.com/jonrobson >> * @rakugojon >> >> _______________________________________________ >> Mobile-l mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > > > > > -- > Dan Duvall > Automation Engineer > Wikimedia Foundation -- Jon Robson * http://jonrobson.me.uk * https://www.facebook.com/jonrobson * @rakugojon _______________________________________________ Mobile-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mobile-l
