On Fri, Jun 6, 2014 at 6:20 PM, James Forrester <[email protected]> wrote:
> On 6 June 2014 16:21, Chris McMahon <[email protected]> wrote: > >> On Fri, Jun 6, 2014 at 4:16 PM, Matthew Flaschen < >> [email protected]> wrote: >> >>> Are they voting or non-voting? >>> >> >> I think browser tests should always be non-voting. Therein lies a >> day-long training session... >> > > And FWIW I fundamentally disagree. :-) > > Non-voting tests are routinely ignored. If your change breaks a test, you > should either fix your code or update the test; both of these actions are > conscious responses to the outcome of the test, and if they're not part of > the pre-merge workflow they too rarely don't happen. > We should probably have a clear understanding about what "voting" means. As I understand it, having an automated test be voting means that a failing test will generate a "-2/DO NOT MERGE" review and prevent a branch from being merged to the master branch. I would encourage a set of browser tests that runs on an automated, highly-controlled, pre-master-branch, non-beta-labs dev environments that might be voting. We don't have that now, but Dan Duvall's current work updating the Vagrant dev environments may lead in that direction. I think the MobileFrontend team has come the closest when they run browser tests against local dev envs, but those local dev envs are maintained by hand and are some way away from being able to be e.g. deployed automatically via Jenkins. The MF folks generally merge new browser tests to master in the same branch as the new feature being tested. Post-merge browser tests look like: Launch a selenium process To reach out over the internet to talk to a browser Running on a VM of some sort To reach out over a different part of the internet To manipulate a shared public test environment like beta labs or test2wiki Whose result is matched to expectations And these tests are intended to expose system and interaction issues as well as code issues in individual features. Also, I encourage thinking along the lines of "if it's a bug in beta labs then it is a bug, period, and we need to fix it". -Chris
_______________________________________________ QA mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/qa
