Re: [Wikitech-l] Browser tests for core

[Antoine Musso]

Le 25/06/2014 00:46, Chris Steipp a écrit :
> I just +2'ed a change to add a few basic selenium tests to core [1]. I
> think it will benefit us all to have a set of automated tests to
> quickly make sure mediawiki is working correctly. From a security
> perspective, this also takes a step towards more efficient security
> testing, which I'm also a fan of (if you've tried blindly scanning
> mediawiki, you know what I'm talking about..).


The next big step would be to trigger them whenever someone propose a
patchset in Gerrit or vote +2.  We have some scripts to setup a
MediaWiki locally and run the browser tests, but that proven to be a bit
unstable :-/

I am wondering how developer would feel with having browser tests
reported back to Gerrit.  Possibly has a different check so we dont have
to wait for them to complete before reporting all the other tests.



> I'd like to see more tests added and backported to REL1_23 to make
> sure we have an ongoing suite to check releases against for next few
> years that we support that LTS. If anyone is interested in both
> mediawiki core and browser tests, I'm sure the QA team would like to
> get you involved.

That is a nice idea.  We could then trigger them before releasing a new
tarball which will add some confidence.  There is a bit of script needed
there but that is definitely doable.

> Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me
> and getting this done. I'll let them jump in with all the details I've
> missed.

Oh I have done nothing beside requesting someone to write the
announcement :-]

-- 
Antoine "hashar" Musso


___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Browser tests for core

[Chris Steipp]

On Jun 24, 2014 6:13 PM, "Dan Garry"  wrote:
>
> On 24 June 2014 17:05, Risker  wrote:
> >
> > Sorry to be a bit OT, but if you guys are going to test, please don't
do it
> > in article space on enwiki, or this is what is going to happen to the
> > accounts.  We've had to almost kick WMF staff off enwiki before because
> > they kept testing in live article space, please don't do that.
> >
>
> Or you'll accidentally insert vandalism into articles when testing the
> abuse filter. Because I've *totally never* done that by accident. :-)
>
> If these browsers tests don't interact with the site that leaves anything
> user-facing behind (e.g. making edits or take actions that create log
> entries), there's no problem with running them on enwiki. Otherwise, they
> should be using our test or beta sites.

They do leave some artifacts (documented in the change, iirc). So yeah,
they should only be used against your dev environment or beta.

>
> Dan
>
> --
> Dan Garry
> Associate Product Manager for Platform and Mobile Apps
> Wikimedia Foundation
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Browser tests for core

[Dan Garry]

On 24 June 2014 17:05, Risker  wrote:
>
> Sorry to be a bit OT, but if you guys are going to test, please don't do it
> in article space on enwiki, or this is what is going to happen to the
> accounts.  We've had to almost kick WMF staff off enwiki before because
> they kept testing in live article space, please don't do that.
>

Or you'll accidentally insert vandalism into articles when testing the
abuse filter. Because I've *totally never* done that by accident. :-)

If these browsers tests don't interact with the site that leaves anything
user-facing behind (e.g. making edits or take actions that create log
entries), there's no problem with running them on enwiki. Otherwise, they
should be using our test or beta sites.

Dan

-- 
Dan Garry
Associate Product Manager for Platform and Mobile Apps
Wikimedia Foundation
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Browser tests for core

[Brian Wolff]

On Jun 24, 2014 9:05 PM, "Risker"  wrote:
>
> On 24 June 2014 18:56, Ryan Kaldari  wrote:
>
> > Nice work guys! One slight issue though. The user Selenium_user (as set
in
> > environment_variables) was just indefinitely blocked on en.wiki :(
> >
> >
>
> Sorry to be a bit OT, but if you guys are going to test, please don't do
it
> in article space on enwiki, or this is what is going to happen to the
> accounts.  We've had to almost kick WMF staff off enwiki before because
> they kept testing in live article space, please don't do that.
>
> Just be glad that the blocking admin didn't make it a block with IP
> autoblocked for 24 hours, or there would have been a bigger problem.
>
> Testwiki is for testand if you must test on enwiki, do it in
userspace.
>
> Risker/Anne
>

That sounds entirely reasonable to me.

--bawolff
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Browser tests for core

[Risker]

On 24 June 2014 18:56, Ryan Kaldari  wrote:

> Nice work guys! One slight issue though. The user Selenium_user (as set in
> environment_variables) was just indefinitely blocked on en.wiki :(
>
>

Sorry to be a bit OT, but if you guys are going to test, please don't do it
in article space on enwiki, or this is what is going to happen to the
accounts.  We've had to almost kick WMF staff off enwiki before because
they kept testing in live article space, please don't do that.

Just be glad that the blocking admin didn't make it a block with IP
autoblocked for 24 hours, or there would have been a bigger problem.

Testwiki is for testand if you must test on enwiki, do it in userspace.

Risker/Anne
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Browser tests for core

[Dan Duvall]

Thanks for reviewing that, Chris!

Chris McMahon did most of the hard work there, porting everything from
qa/browsertests into core. I was glad to help on the Vagrant side of
things, though: I have a mediawiki-vagrant commit standing by that will do
all the heavy heavy installation lifting that you mentioned. :) I'll double
check that it's backwards compatible and send it up for review shortly.



On Tue, Jun 24, 2014 at 3:46 PM, Chris Steipp  wrote:

> I just +2'ed a change to add a few basic selenium tests to core [1]. I
> think it will benefit us all to have a set of automated tests to
> quickly make sure mediawiki is working correctly. From a security
> perspective, this also takes a step towards more efficient security
> testing, which I'm also a fan of (if you've tried blindly scanning
> mediawiki, you know what I'm talking about..).
>
> I think the QA group is working on vagrant-izing these, but if you
> have ruby >1.9.3 and firefox, then setting up and running these tests
> on your local dev system is 4-6 commands,
>
> $ cd tests/browser
> $ gem update --system
> $ gem install bundler
> $ bundle install
>
> You can either set your environment variables yourself, or edit
> environment_variables and run `source environment_variables` to set
> them. Then it's just
>
> $ bundle exec cucumber features/
>
> to run the tests. They currently complete in 36 seconds on my laptop.
>
> I'd like to see more tests added and backported to REL1_23 to make
> sure we have an ongoing suite to check releases against for next few
> years that we support that LTS. If anyone is interested in both
> mediawiki core and browser tests, I'm sure the QA team would like to
> get you involved.
>
> Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me
> and getting this done. I'll let them jump in with all the details I've
> missed.
>
>
> [1] - https://gerrit.wikimedia.org/r/#/c/133507/
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l




-- 
Dan Duvall
Automation Engineer
Wikimedia Foundation 
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Browser tests for core

[Chris McMahon]

On Tue, Jun 24, 2014 at 3:56 PM, Ryan Kaldari 
wrote:

> Nice work guys! One slight issue though. The user Selenium_user (as set in
> environment_variables) was just indefinitely blocked on en.wiki :(
>

Yes, and I think we should keep it that way. See my msg. to the mobile-l
and qa lists earlier today:
http://lists.wikimedia.org/pipermail/mobile-l/2014-June/007435.html

These tests are intended as acceptance tests for a new mediawiki install
and smoke tests for a test env like beta labs.  Let's not run them against
production.

-Chris



>
> Ryan Kaldari
>
>
> On Tue, Jun 24, 2014 at 3:46 PM, Chris Steipp 
> wrote:
>
> > I just +2'ed a change to add a few basic selenium tests to core [1]. I
> > think it will benefit us all to have a set of automated tests to
> > quickly make sure mediawiki is working correctly. From a security
> > perspective, this also takes a step towards more efficient security
> > testing, which I'm also a fan of (if you've tried blindly scanning
> > mediawiki, you know what I'm talking about..).
> >
> > I think the QA group is working on vagrant-izing these, but if you
> > have ruby >1.9.3 and firefox, then setting up and running these tests
> > on your local dev system is 4-6 commands,
> >
> > $ cd tests/browser
> > $ gem update --system
> > $ gem install bundler
> > $ bundle install
> >
> > You can either set your environment variables yourself, or edit
> > environment_variables and run `source environment_variables` to set
> > them. Then it's just
> >
> > $ bundle exec cucumber features/
> >
> > to run the tests. They currently complete in 36 seconds on my laptop.
> >
> > I'd like to see more tests added and backported to REL1_23 to make
> > sure we have an ongoing suite to check releases against for next few
> > years that we support that LTS. If anyone is interested in both
> > mediawiki core and browser tests, I'm sure the QA team would like to
> > get you involved.
> >
> > Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me
> > and getting this done. I'll let them jump in with all the details I've
> > missed.
> >
> >
> > [1] - https://gerrit.wikimedia.org/r/#/c/133507/
> >
> > ___
> > Wikitech-l mailing list
> > Wikitech-l@lists.wikimedia.org
> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Browser tests for core

[Greg Grossmeier]

> Nice work guys! One slight issue though. The user Selenium_user (as set in
> environment_variables) was just indefinitely blocked on en.wiki :(

That shouldn't affect these tests. These run on your local box/vagrant.

-- 
| Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @gregA18D 1138 8E47 FAC8 1C7D |

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Browser tests for core

[Ryan Kaldari]

Nice work guys! One slight issue though. The user Selenium_user (as set in
environment_variables) was just indefinitely blocked on en.wiki :(

Ryan Kaldari


On Tue, Jun 24, 2014 at 3:46 PM, Chris Steipp  wrote:

> I just +2'ed a change to add a few basic selenium tests to core [1]. I
> think it will benefit us all to have a set of automated tests to
> quickly make sure mediawiki is working correctly. From a security
> perspective, this also takes a step towards more efficient security
> testing, which I'm also a fan of (if you've tried blindly scanning
> mediawiki, you know what I'm talking about..).
>
> I think the QA group is working on vagrant-izing these, but if you
> have ruby >1.9.3 and firefox, then setting up and running these tests
> on your local dev system is 4-6 commands,
>
> $ cd tests/browser
> $ gem update --system
> $ gem install bundler
> $ bundle install
>
> You can either set your environment variables yourself, or edit
> environment_variables and run `source environment_variables` to set
> them. Then it's just
>
> $ bundle exec cucumber features/
>
> to run the tests. They currently complete in 36 seconds on my laptop.
>
> I'd like to see more tests added and backported to REL1_23 to make
> sure we have an ongoing suite to check releases against for next few
> years that we support that LTS. If anyone is interested in both
> mediawiki core and browser tests, I'm sure the QA team would like to
> get you involved.
>
> Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me
> and getting this done. I'll let them jump in with all the details I've
> missed.
>
>
> [1] - https://gerrit.wikimedia.org/r/#/c/133507/
>
> ___
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Browser tests for core

[James Forrester]

On 24 June 2014 15:46, Chris Steipp  wrote:

> I just +2'ed a change to add a few basic selenium tests to core [1]. I
> think it will benefit us all to have a set of automated tests to
> quickly make sure mediawiki is working correctly.


​Great news; thank you all!

J.
-- 
James D. Forrester
Product Manager, Editing
Wikimedia Foundation, Inc.

jforres...@wikimedia.org | @jdforrester
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Browser tests for core

[Chris Steipp]

I just +2'ed a change to add a few basic selenium tests to core [1]. I
think it will benefit us all to have a set of automated tests to
quickly make sure mediawiki is working correctly. From a security
perspective, this also takes a step towards more efficient security
testing, which I'm also a fan of (if you've tried blindly scanning
mediawiki, you know what I'm talking about..).

I think the QA group is working on vagrant-izing these, but if you
have ruby >1.9.3 and firefox, then setting up and running these tests
on your local dev system is 4-6 commands,

$ cd tests/browser
$ gem update --system
$ gem install bundler
$ bundle install

You can either set your environment variables yourself, or edit
environment_variables and run `source environment_variables` to set
them. Then it's just

$ bundle exec cucumber features/

to run the tests. They currently complete in 36 seconds on my laptop.

I'd like to see more tests added and backported to REL1_23 to make
sure we have an ongoing suite to check releases against for next few
years that we support that LTS. If anyone is interested in both
mediawiki core and browser tests, I'm sure the QA team would like to
get you involved.

Big thanks to hashar, Chris McMahon, and Dan Duvall for indulging me
and getting this done. I'll let them jump in with all the details I've
missed.


[1] - https://gerrit.wikimedia.org/r/#/c/133507/

___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l