Awesome stuff. On Wed, Jun 24, 2015 at 8:25 PM, Jon Robson <[email protected]> wrote:
> I haven't been able to use verified yet - I think this is a permission > that Barry doesn't have. > I'm experimenting with him doing no score reviews right now rather > than +1s for good patches. The -1s are worth being as visible as > possible. > > I'm hoping to send an update on the status quo but I've updated him so > that he can be run in a script: > > https://github.com/jdlrobson/Barry-the-Browser-Test-Bot/blob/master/barrybot.py > If you want him to re-review code simply remove him from the list of > reviewers - the bot currently works out what it needs to review based > on which patches he is currently not a reviewer for. > > > On Wed, Jun 24, 2015 at 5:36 AM, [email protected] > <[email protected]> wrote: > >> How does it affect jenkins if something/somebody sets verified to -1 for > >> example? Would it block it from merging? > > > > > > > > Any -1/-2 in the verified column blocks submitting a change (even, if > other > > reviewers add a +1 in verified), so if Barry wouldn't be happy, you can't > > merge. A workaround is to remove BarryBot as a reviewer (which would > remove > > the vote, too, and allows jenkins to submit the change). > > > > > > > > Freundliche Grüße > > Florian Schmidt > > > > > > > > > > > > > > > > -----Original-Nachricht----- > > > > Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch > > > > Datum: Wed, 24 Jun 2015 14:07:23 +0200 > > > > Von: Joaquin Oltra Hernandez <[email protected]> > > > > An: "[email protected]" > > <[email protected]> > > > > > > > > > > > > > > > > Baha I agree with you, but instead of staying silent just commenting a > > "BarryBot is happy! Browser tests passed" without setting any +1 would be > > good to know they run. > > > > I also agree with Florian, if we could set Verified +1 or -1 that could > be > > interesting. How does it affect jenkins if something/somebody sets > verified > > to -1 for example? Would it block it from merging? > > > > On Wed, Jun 24, 2015 at 1:53 PM, [email protected] > > <[email protected]> wrote: > >> > >> Or, maybe, it should be possible for the bot to set the verified flag, > >> instead of the code review (it's not a code review, so a minus one is > >> misleading, too). > >> > >> > >> > >> Freundliche Grüße > >> Florian Schmidt > >> > >> > >> > >> > >> > >> > >> > >> -----Original-Nachricht----- > >> > >> Betreff: Re: [WikimediaMobile] [Update] Browser tests per patch > >> > >> Datum: Wed, 24 Jun 2015 13:47:39 +0200 > >> > >> Von: Bahodir Mansurov <[email protected]> > >> > >> An: Jon Robson <[email protected]> > >> > >> > >> > >> > >> > >> > >> > >> May I suggest that Barry should not +1 patches. It gives a false > >> impression that the code is alright even though the code may not be > covered > >> at all. Can we have it just -1 when there is a problem, and stay silent > >> otherwise? > >> > >> > >> On Jun 18, 2015, at 8:55 PM, Jon Robson <[email protected]> wrote: > >> > >> So far I'm seeing some extremely positive results. The tests are > >> running super fast (in phantomjs the smoke tests are taking less than > >> 3 minutes...). > >> > >> I've been manually running him whilst I do code review in parallel and > >> he's already generated some interesting conversation on this patchset: > >> https://gerrit.wikimedia.org/r/#/c/219249/ > >> > >> If you want to pair and get this to be less hacky I'm more than happy! > >> > >> > >> On Thu, Jun 18, 2015 at 5:41 PM, Dan Duvall <[email protected]> > wrote: > >> > >> Nice work, Jon. > >> > >> I've opened a task for defining a JJB builder/template and getting > >> something > >> like this into CI sooner rather than later.[1] I think your setup proves > >> that a set of well-groomed MW-Selenium integration tests can be stable > >> enough for this purpose, and we can start with an even smaller subset of > >> core tests for a pre-merge build. Of course this isn't something that we > >> planned to do 'now' now—sometimes 'then' suddenly becomes 'now' so > >> 'soon'—but we can start with an experiment on Gather or MobileFrontend > >> tests, since their health has greatly improved, and see how it goes. > >> > >> [1] https://phabricator.wikimedia.org/T103039 > >> > >> On Thu, Jun 18, 2015 at 11:03 AM, Jon Robson <[email protected]> > wrote: > >> > >> > >> So the script that actually runs the browser test is not in a generic > >> useful form but it is: > >> https://gist.github.com/jdlrobson/32b607f8009e897ee80c > >> > >> It uses the GerritCommandLine tool to do grabbing and reviewing > >> https://github.com/jdlrobson/GerritCommandLine > >> > >> Ideally if we can use labs-tools-gerrit-to-redis for identifying > >> patches and then pulling them down we wouldn't need the > >> GerritCommandLine tool since the code to submit a review is pretty > >> trivial and captured in this function: > >> > https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277 > >> > >> I've also put this in the task > >> https://phabricator.wikimedia.org/T101069#1379462 > >> > >> We'll probably want an instance per extension, to simplify having to > >> worry about dependencies (we can just run a git update on all > >> extensions after each checkout) > >> > >> On Thu, Jun 18, 2015 at 4:32 AM, Joaquin Oltra Hernandez > >> <[email protected]> wrote: > >> > >> Awesome Jon! I'm so happy to finally see this developing :DD > >> > >> Loving the : `Browserbot happy!` > >> > >> I've noticed it can report either the name of the failing test or the > >> full > >> log. What do you think if we show that, and a url with the pasted log > >> somewhere publicly to not put too much noise on the comments but still > >> be > >> able to see it? Something like > >> https://phabricator.wikimedia.org/paste/... > >> > >> +1 to where is the source. > >> +1 to documenting how you've set it all up on wiki somewhere. > >> > >> I also think we need a catchy phrase for the -1s! > >> > >> Thanks for you work on this, we'll get more focused time for it soon. > >> > >> On Thu, Jun 18, 2015 at 11:33 AM, Sam Smith <[email protected]> > >> wrote: > >> > >> > >> I agree with Florian everything that you've written should be in a > >> public > >> version control system. > >> > >> Second, I'd ask that you document your experiences so far in getting > >> this > >> set up and how it works so that other members of the vertical can help > >> to > >> maintain it moving forward. > >> > >> Third, great work!!1 > >> > >> <3 > >> > >> –Sam > >> > >> On Thu, Jun 18, 2015 at 7:09 AM, [email protected] > >> <[email protected]> wrote: > >> > >> > >> > >> It's currently working via a script that you can find here: > >> /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh > >> > >> > >> It would be great to have the script in a public version control > >> system > >> (e.g. github?), especially for people, e.g. volunteers, who can't ssh > >> to > >> gather-browser-tests.eqiad.wmflabs[1] > >> > >> [1] all people, who're not members of > >> https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile-smoketests > >> > >> Best, > >> Florian > >> > >> -----Original-Nachricht----- > >> Betreff: [WikimediaMobile] [Update] Browser tests per patch > >> Datum: Thu, 18 Jun 2015 03:27:32 +0200 > >> Von: Jon Robson <[email protected]> > >> An: "QA (software quality assurance) for Wikimedia projects." > >> <[email protected]>, mobile-l <[email protected]> > >> > >> Background: mobile wants to gain more confidence in its browser tests > >> by running a subset of browser tests on a case by case basis [0]. > >> > >> Good news: I've got a proof of concept running and Barry the browser > >> test bot has given some legitimate helpful reviews to Gather [1]. > >> > >> Even better news: It's proving itself valuable already [2]. > >> As you can see in the messages the bot has posted on [3] we have a > >> couple of options on display option format for his reviews. > >> > >> So.. hopefully this short experience has sold you all already. > >> > >> This script is currently a manual job and needs a bit of tweaking > >> before we can put it in a cron job/run it always - it needs to watch > >> for new commits and then run a modification of the above script on a > >> per case basis (if two versions of it run in parallel we have an > >> issue). > >> > >> Definitely something we should push for next sprint! > >> > >> Long live Barry bot! > >> > >> Devs... (everyone else now of what follows is likely to be useful): > >> I got the labs instance up and running on: > >> http://gather-browser-tests.wmflabs.org/wiki/Main_Page > >> > >> Most of you in readership team should be able to ssh > >> gather-browser-tests.eqiad.wmflabs > >> Let me know if you have no access. > >> > >> It's currently working via a script that you can find here: > >> /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh > >> > >> [0] https://phabricator.wikimedia.org/T100293 > >> [1] > >> > >> > >> > https://gerrit.wikimedia.org/r/#/q/reviewer:jdlrobson%252Bbarry%2540gmail.com+status:open,n,z > >> [2] https://gerrit.wikimedia.org/r/#/c/218731/ > >> > >> _______________________________________________ > >> Mobile-l mailing list > >> [email protected] > >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >> > >> > >> > >> > >> > >> > >> _______________________________________________ > >> Mobile-l mailing list > >> [email protected] > >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >> > >> > >> > >> > >> _______________________________________________ > >> Mobile-l mailing list > >> [email protected] > >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >> > >> > >> > >> _______________________________________________ > >> Mobile-l mailing list > >> [email protected] > >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >> > >> > >> > >> > >> -- > >> Jon Robson > >> * http://jonrobson.me.uk > >> * https://www.facebook.com/jonrobson > >> * @rakugojon > >> > >> _______________________________________________ > >> Mobile-l mailing list > >> [email protected] > >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >> > >> > >> > >> > >> > >> -- > >> Dan Duvall > >> Automation Engineer > >> Wikimedia Foundation > >> > >> > >> > >> > >> -- > >> Jon Robson > >> * http://jonrobson.me.uk > >> * https://www.facebook.com/jonrobson > >> * @rakugojon > >> > >> _______________________________________________ > >> Mobile-l mailing list > >> [email protected] > >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >> > >> > >> _______________________________________________ > >> Mobile-l mailing list > >> [email protected] > >> https://lists.wikimedia.org/mailman/listinfo/mobile-l > >> > > > > > > -- > Jon Robson > * http://jonrobson.me.uk > * https://www.facebook.com/jonrobson > * @rakugojon >
_______________________________________________ Mobile-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mobile-l
