On Wed, Nov 3, 2010 at 5:06 AM, Jonathan Lange <[email protected]> wrote: > On Wed, Nov 3, 2010 at 2:01 AM, Deryck Hodge <[email protected]> > wrote: >> We don't know the value of these tests or how much usefulness they >> provide. We don't know the amount of coverage we actually get from >> them. We have half or better of a four+ hour test run for these tests >> and we have no way to assess the value of this cost. We have >> developers avoiding page tests and windmill at all costs and no clear >> story of even why we use them or what we expect from them. It seems >> to me a short and controlled experiment is an easy way to get some >> data on these questions. >> > > Would you not get the same data by running the page & windmill tests > separately as Gary suggests? >
Yes, you could certainly get data this way. And as I said in Francis' email, if people really prefer this approach, then I'll withdraw my proposal. I'm not personally interested in championing what Gary suggests, not because I don't see some value over what we have now, but because I'm just not as passionate about it nor do I feel the benefit is worth the effort. I'm skeptical that we'll ever make a determination about the value or benefit of these tests once we set up another builder, mostly because by setting up the builder there is an explicit assertion that these tests have value and we shouldn't lose them. So I imagine a year from now these tests will be blocking deployments, and people will be complaining the same as they do now for landings, but we'll still be reluctant to change anything because we assume test failure implies value. Also, part of my goal with disabling page and windmill tests is to get us looking at the UI more in QA. If the tests are run before deployment, there is no incentive to really examine the UI and user interactions during QA. The tests provide false assurance of UI quality, and I don't see this changing if we set up another builder. Cheers, deryck -- Deryck Hodge https://launchpad.net/~deryck http://www.devurandom.org/ _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
