<quote name="Martijn Hoekstra" date="2014-03-10" time="08:51:38 +0100"> > If the test infrastructure can't handle running every test on every commit > (why can't they by the way, and could that be remedied?) Would it be > possible and desireable to make sure a more elaborate test suite with all > available tests in run as a gate before cutting the production branch?
I'm hesitant on making a suite of tests that only run once a week be a hard go/no-go. I'm hesitant because that won't help us actually create better code. Sure, it'll prevent broken code from going out, but we're already pretty good about that (since we only deploy to testwikis initially and find breakages before it goes out to 'real' (minus mediawiki.org) wikis). Put another way, the breakages that the tests find are a subset of what Reedy finds and fixes/reverts when he deploys to the testwikis. And the breakages that somehow make it to 'real' wikis are by definition, not caught by the test suite (as the test suite has already run against production wikis). But it won't make the developers writing bad code write better code. The only that that will do that is Jenkins reporting back into Gerrit "You shall not pass!". And the only way of being able to do that is if we (Jenkins) knows which change caused the breakage. That means per-commit tests. Not weekly tests. We already have twice-daily tests running (and those are the ones that cause the change that started this thread) because they can't be run for every commit. I'd rather not just pick a random one of those (effectively) and say "you're more important that the other identical runs of yourself". Feedback. Developer feedback is the only way to make this better. The solution, honestly, is making all tests run per commit. https://upload.wikimedia.org/wikipedia/en/7/74/Continuous_Delivery_process_diagram.png s/Delivery Team/Developers/ and that's what should happen for every commit. This isn't easy, and it is what we (the WMF Release Engineering & QA Team) are working on. Greg -- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D | _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
