On Wed, Jul 1, 2009 at 12:28 PM, Frank Schönheit - Sun Microsystems Germany<[email protected]> wrote: >> Autotests are our best weapon against regressions. >> But only if the right ones get written and run. > > And if > - they are reliable and reproducible > - run in reasonable time > > Please allow me my slight doubts that all of our autotest-frameworks > (and we have at least 5: cwscheckapi, complex test cases, testtool, > performance tests, confwatch tests) fulfill both requirements. (And yes, > some of them don't even fulfill the "they're run" requirement you named.)
It's definitely a struggle keeping a large set of tests reliable enough that failing one test is enough to close the tree. Every day, a different pair of developers is assigned to be Build Sheriff. Sheriffs close the tree when it's too red, and do whatever it takes to get the tree green again. It's also a big effort rounding up enough machines to run the tests with any sort of reasonable cycle time. Chromium has about fifty machines running tests continuously against the trunk (see http://build.chromium.org ) and another twenty running precommit tests (see http://build.chromium.org/buildbot/try-server/ ). > If we all agree that auto tests are highly useful, and an important line > of defense against bugs creeping into a release branch, then I wonder > why we invest so little resources into making our QA frameworks fulfill > those 3 basic and essential requirements. > > I'd *love* to see that changed. Heck, I'm a developer, writing tests is > as boring as anything :), but it could even be *fun* writing UI tests > (like currently done with testtool scripts) if it would happen in Java, > with a decent IDE ... FWIW, Chromium doesn't have an IDE for writing UI tests. - Dan --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
