On Tue, Nov 2, 2010 at 1:54 PM, Francis J. Lacoste <[email protected]> wrote: > > On November 2, 2010, Deryck Hodge wrote: >> On Tue, Nov 2, 2010 at 12:48 PM, Robert Collins >> >> <[email protected]> wrote: >> > On Wed, Nov 3, 2010 at 6:44 AM, Deryck Hodge <[email protected]> > wrote: >> >> On Tue, Nov 2, 2010 at 12:33 PM, Robert Collins >> >> >> >> <[email protected]> wrote: >> >>> 2) Whats the rollback strategy? >> >> >> >> I see this as "re-enable the tests and go back to what we have now." >> >> I feel like that is too simple an answer for your question, so maybe >> >> I'm missing something that your concerned about? >> > >> > Its just that tests tend to bitrot when they aren't run. It may be >> > hard to turn them back on. >> >> Yeah, that's a valid concern. I don't see anyway to avoid that risk. >> I don't think 3 months is so long that the bitrot will be that huge >> should this prove a failure, but that's just a gut feeling and I have >> little to base that on, save how often page and windmill tests seem to >> be changed normally. Usually the changes are simple string >> substitutions. I, of course, would be willing to take on this pain >> and fix the tests since I'm the one proposing the experiment. >> > > I think making sure that the experiment is easy to revert is a very valid > concern. > > I would suggest we do it differently then to ensure that possibility: > > Stop running pagetests/windmill pre-commit. Move them to a standalone buildbot > that runs them exclusively, as much as it can. That buildbot blocks > deployment, so no deployment of a stretch of QA revisions until a clean > windmill/pagetests is blessed. > > I think this has several benefits: > > a- Ensure that the tests don't bitrot. > b- Enforces the requirements of moving pagetest coverage into unit test > coverage. > c- Still speeds up the landing of code (altough probably doesn't speed up the > deployment of code as much.) > d- Makes the cost/benefit evaluation of the change more obvious: > - Classify the failures on the integration buildbot into: > - Instability failure > - Regression (failed windmill or page test) > > During the experiment, we shouldn't add any new windmill or pagetest. At the > end of the evaluation period, we should look at the data in d, argue over the > relative weight of loss of time due to instability failure vs regression and > either: keep that new setup, revert to the old one, drop everything and move > forward.
This feel like a different experiment than I'm proposing. There's a lot in that to save tests that *might* be useful, and for the course of the experiment, doesn't leave us in a markedly different state than we are today. If everyone were standing up saying "these tests have saved us time and again, what are you thinking?!?!" I could understand this reaction. At the point we start getting into alternate builders and testing landings vs. testing deployments that gets into more than I can reasonably champion, i.e. it's a foundations dev story not a process experiment. I'll happily make changes to the process I proposed if there are changes I can make that keep this process driven but make people feel better about my proposal, but I really don't want to take on a build environment dev story. Of course, if everyone feels better about this alternate testing story, then I'm happy to defer to that and drop the proposal. Cheers, deryck -- Deryck Hodge https://launchpad.net/~deryck http://www.devurandom.org/ _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
