On 11/25/2013 04:23 AM, Clint Byrum wrote: > Excerpts from Joe Gordon's message of 2013-11-24 21:00:58 -0800: >> Hi All, >> >> TL;DR Last week the gate got wedged on nondeterministic failures. Unwedging >> the gate required drastic actions to fix bugs. >> >> > <snip> > > (great write-up, thank you for the details, and thank you for fixing > it!) > >> >> Now that we have the gate back into working order, we are working on the >> next steps to prevent this from happening again. The two most immediate >> changes are: >> >> - Doing a better job of triaging gate bugs ( >> >> http://lists.openstack.org/pipermail/openstack-dev/2013-November/020048.html >> ). >> >> >> - In the next few days we will remove 'reverify no bug' (although you >> will still be able to run 'reverify bug x'. >> > > I am curious, why not also disable 'recheck no bug'?
recheck no bug still has a host of valid use cases. Often times I use it when I upload a patch, it fails because of a thing somewhere else, we fix that, and I need to recheck the patch because it should work now. It's also not nearly as dangerous as reverify no bug. > I see this as a failure of bug triage. A bug that has more than 1 > recheck/reverify attached to it is worth a developer's time. The data > gathered through so many test runs is invaluable when chasing races like > the ones that cause these intermittent failures. If every core dev of > every project spent 10 working minutes every day looking at the rechecks > page to see if there is an untriaged recheck there, or just triaging bugs > in general, I suspect we'd fix these a lot quicker. > > I do wonder if we would be able to commit enough resources to just run > two copies of the gate in parallel each time and require both to pass. > Doubling the odds* that we will catch an intermittent failure seems like > something that might be worth doubling the compute resources used by > the gate. Funny story- there is a patch coming to do just that. > *I suck at math. Probably isn't doubling the odds. Sounds > good though. ;) > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
