For data points: In the last successful build:
Page tests: 50m (~20%) Bugs windmill: 10m (~4%) Code windmill: 5m (~2%) Registry windmill: 7m (~3%) Soyuz windmill: 1m (0%) Translations windmill: 8m ((~3%) Windmill + page tests: (~32%) Total tests run time: 254m (100%) So we are talking about a third of the test suite time taken up by the page and windmill tests. -- Francis J. Lacoste [email protected] On November 3, 2010, Guilherme Salgado wrote: > On Tue, 2010-11-02 at 08:59 -0500, Deryck Hodge wrote: > > Hi, all. > > > > I have some thoughts about our page and windmill tests, and I'd like > > to propose an experiment related to these tests. This is inspired by > > my own thinking about how to best test web UI and by Rob's recent > > cost/benefit analysis arguments for optional reviews. > > > > Here's my assessment of the problem: > > > > The burden we carry for our current page tests and Windmill tests does > > not match the benefit we get from them. > > > > The burden: > > > > * Much longer test run times (the bugs module goes from 45 minutes to > > 20 locally without them) > > * Fragile tests that block landings > > * Fragile infrastructure (see issues with Windmill tests under load) > > * Confusion over how to best test UI (page tests vs. integration vs. > > browser unit tests) > > Although most of the above is true for page tests, I think it's a lot > more problematic for windmill tests (specially the run time and fragile > infrastructure), so it may be worth considering your experiment just for > windmill tests? > > I'd expect most of the 25 minutes saved by disabling page/windmill tests > would be spent in the windmill ones, but maybe you already know how long > it takes to run the tests under lp.app.bugs disabling just the windmill > tests?
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
