Re: [openstack-dev] Unwedging the gate

2013-11-26 Thread Sean Dague
On 11/25/2013 09:14 PM, James E. Blair wrote: > Joe Gordon writes: > >> On Sun, Nov 24, 2013 at 10:48 PM, Robert Collins >> wrote: >> >>> On 25 November 2013 19:25, Joe Gordon wrote: On Sun, Nov 24, 2013 at 9:58 PM, Robert Collins < >>> robe...@robertcollins.net> wrote:

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread James E. Blair
Joe Gordon writes: > On Sun, Nov 24, 2013 at 10:48 PM, Robert Collins > wrote: > >> On 25 November 2013 19:25, Joe Gordon wrote: >> > >> > >> > >> > On Sun, Nov 24, 2013 at 9:58 PM, Robert Collins < >> robe...@robertcollins.net> >> > wrote: >> >> >> >> I have a proposal - I think we should mark

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Joe Gordon
On Sun, Nov 24, 2013 at 10:48 PM, Robert Collins wrote: > On 25 November 2013 19:25, Joe Gordon wrote: > > > > > > > > On Sun, Nov 24, 2013 at 9:58 PM, Robert Collins < > robe...@robertcollins.net> > > wrote: > >> > >> I have a proposal - I think we should mark all recheck bugs critical, > >> and

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Sean Dague
On 11/25/2013 11:54 AM, Clint Byrum wrote: > Excerpts from Robert Collins's message of 2013-11-25 01:30:11 -0800: >> On 25 November 2013 22:23, Clint Byrum wrote: >> >>> I do wonder if we would be able to commit enough resources to just run >>> two copies of the gate in parallel each time and requ

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Jeremy Stanley
On 2013-11-25 09:55:03 -0800 (-0800), Clint Byrum wrote: > Excerpts from Monty Taylor's message of 2013-11-25 06:52:02 -0800: [...] > > recheck no bug still has a host of valid use cases. Often times > > I use it when I upload a patch, it fails because of a thing > > somewhere else, we fix that, an

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Clint Byrum
Excerpts from Monty Taylor's message of 2013-11-25 06:52:02 -0800: > > On 11/25/2013 04:23 AM, Clint Byrum wrote: > > Excerpts from Joe Gordon's message of 2013-11-24 21:00:58 -0800: > >> Hi All, > >> > >> TL;DR Last week the gate got wedged on nondeterministic failures. Unwedging > >> the gate re

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Joshua Harlow
+2 Sent from my really tiny device... > On Nov 25, 2013, at 5:02 AM, "Davanum Srinivas" wrote: > > Many thanks to everyone who helped with the many fixes. Kudos to > Joe/Clark for spear heading the effort! > > -- dims > >> On Mon, Nov 25, 2013 at 12:00 AM, Joe Gordon wrote: >> Hi All, >> >>

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Clint Byrum
Excerpts from Robert Collins's message of 2013-11-25 01:30:11 -0800: > On 25 November 2013 22:23, Clint Byrum wrote: > > > I do wonder if we would be able to commit enough resources to just run > > two copies of the gate in parallel each time and require both to pass. > > Doubling the odds* that

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Monty Taylor
On 11/25/2013 04:23 AM, Clint Byrum wrote: > Excerpts from Joe Gordon's message of 2013-11-24 21:00:58 -0800: >> Hi All, >> >> TL;DR Last week the gate got wedged on nondeterministic failures. Unwedging >> the gate required drastic actions to fix bugs. >> >> > > > (great write-up, thank you for

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Davanum Srinivas
Many thanks to everyone who helped with the many fixes. Kudos to Joe/Clark for spear heading the effort! -- dims On Mon, Nov 25, 2013 at 12:00 AM, Joe Gordon wrote: > Hi All, > > TL;DR Last week the gate got wedged on nondeterministic failures. Unwedging > the gate required drastic actions to fi

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Robert Collins
On 25 November 2013 22:23, Clint Byrum wrote: > I do wonder if we would be able to commit enough resources to just run > two copies of the gate in parallel each time and require both to pass. > Doubling the odds* that we will catch an intermittent failure seems like > something that might be wort

Re: [openstack-dev] Unwedging the gate

2013-11-25 Thread Clint Byrum
Excerpts from Joe Gordon's message of 2013-11-24 21:00:58 -0800: > Hi All, > > TL;DR Last week the gate got wedged on nondeterministic failures. Unwedging > the gate required drastic actions to fix bugs. > > (great write-up, thank you for the details, and thank you for fixing it!) > > Now th

Re: [openstack-dev] Unwedging the gate

2013-11-24 Thread Robert Collins
On 25 November 2013 19:25, Joe Gordon wrote: > > > > On Sun, Nov 24, 2013 at 9:58 PM, Robert Collins > wrote: >> >> I have a proposal - I think we should mark all recheck bugs critical, >> and the respective project PTLs should actively shop around amongst >> their contributors to get them fixed

Re: [openstack-dev] Unwedging the gate

2013-11-24 Thread Joe Gordon
On Sun, Nov 24, 2013 at 9:58 PM, Robert Collins wrote: > I have a proposal - I think we should mark all recheck bugs critical, > and the respective project PTLs should actively shop around amongst > their contributors to get them fixed before other work: we should > drive the known set of nondeter

Re: [openstack-dev] Unwedging the gate

2013-11-24 Thread Robert Collins
I have a proposal - I think we should mark all recheck bugs critical, and the respective project PTLs should actively shop around amongst their contributors to get them fixed before other work: we should drive the known set of nondeterministic issues down to 0 and keep it there. -Rob On 25 Novemb

[openstack-dev] Unwedging the gate

2013-11-24 Thread Joe Gordon
Hi All, TL;DR Last week the gate got wedged on nondeterministic failures. Unwedging the gate required drastic actions to fix bugs. Starting on November 15th, gate jobs have been getting progressively less stable with not enough attention given to fixing the issues, until we got to the point where