We're still working through kinks in the new system, which is why it's
not fully documented yet.

During the 2 weeks of gate wedging in January, we discovered a number of
interesting things. Some review teams are really slow at dealing with
patches with a +2 already on them, and may wait 2 - 6 *weeks* before a
second core reviewer shows up to approve. In that period of time the
dependencies used in the tests probably changed 3 or 4 times. Which
means the test results were invalid.

We also found that once a patch gets a +A people (core and non-core)
blindly reverify on any failure, over and over and over again. I saw one
core developer "reverify 123456789" on a patch.

Or that when trying to land a patch people will recheck half a dozen
times to get one clean run, then get their patch pushed to the gate.

All this blind meatgrinder behavior was putting tons of code into the
gate that could not pass. That, coupled with other race conditions we
were dealing with, put us into a state where we had a 60hr gate queue.

So we're trying out a new system. A change will not move into the gate
unless there is both a +A and Jenkins +1 on the patch.

We also rerun check results on comment if the test results are > 72hrs
old, so there is always a reasonably fresh version of the results. This
also helps detect merge conflicts early.

Also when you +A a patch, if there aren't 24hr fresh test results, we
rerun check first, then if that passes, it gets sent to the gate.

There is still a bit of a sticky part of the flow if it fails in the
gate. Because that means you don't satisfy the +1 jenkins requirement.
Still trying to get the right flow worked out there.

        -Sean

On 02/18/2014 07:51 AM, Sergey Lukjanov wrote:
> You already can reverify on any approved change. Jenkins automatically
> runs check pipeline jobs after 3 days.
> 
> 
> On Tue, Feb 18, 2014 at 4:29 PM, Christopher Yeoh <cbky...@gmail.com
> <mailto:cbky...@gmail.com>> wrote:
> 
>     On Tue, Feb 18, 2014 at 12:27 AM, Gary Kotton <gkot...@vmware.com
>     <mailto:gkot...@vmware.com>> wrote:
> 
>         Thanks!
>         That makes sense. Just odd how a -1 was received.
> 
> 
>     So I think that now a "check" run is automatically done if the
>     latest Jenkins run is too old (7 days?)
>     This is done because gate failures are so costly. The check run
>     failing would return -1 instead of a
>     gate failure of -2.
> 
> 
>      
> 
>         On 2/17/14 3:15 PM, "Akihiro Motoki" <mot...@da.jp.nec.com
>         <mailto:mot...@da.jp.nec.com>> wrote:
> 
>         >Hi Gary,
>         >
>         >According to zuul layout.yaml [1], "reverify bug #" should
>         still work
>         >but it seems to work only when verified score from jenkins is -2.
>         
> >https://urldefense.proofpoint.com/v1/url?u=https://github.com/openstack-in
>         
> >fra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml%23
>         
> >L25&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg4
>         
> >5MkPhCZFxPEq8%3D%0A&m=Nf6%2FyHMSSchQO59xIcg6NKX1O2wlkkHHd42bYQim0k0%3D%0A&
>         >s=8fc4d7e6970936977dc85989e4e7e9429afb15f5091a768efa4ce236417d9c9b
>         >
>         >Note that core team can trigger gate jobs by re-approving the
>         patch.
>         >
>         >Thanks,
>         >Akihiro
>         >
>         >(2014/02/17 22:03), Gary Kotton wrote:
>         >> Hi,
>         >> The "reverify no bug" was removed. But "reverify bug #" used
>         to work.
>         >>That no longer does. With the constant gate failures how can
>         we ensure
>         >>that a approved patch does a retry?
>         >> Thanks
>         >> Gary
>         >>
>         >> From: Sylvain Bauza <sylvain.ba...@gmail.com
>         <mailto:sylvain.ba...@gmail.com>
>         >><mailto:sylvain.ba...@gmail.com <mailto:sylvain.ba...@gmail.com>>>
>         >> Reply-To: "OpenStack Development Mailing List (not for usage
>         >>questions)" <openstack-dev@lists.openstack.org
>         <mailto:openstack-dev@lists.openstack.org>
>         >><mailto:openstack-dev@lists.openstack.org
>         <mailto:openstack-dev@lists.openstack.org>>>
>         >> Date: Monday, February 17, 2014 2:53 PM
>         >> To: "OpenStack Development Mailing List (not for usage
>         questions)"
>         >><openstack-dev@lists.openstack.org
>         <mailto:openstack-dev@lists.openstack.org>
>         >><mailto:openstack-dev@lists.openstack.org
>         <mailto:openstack-dev@lists.openstack.org>>>
>         >> Subject: Re: [openstack-dev] [infra] reverify/recheck
>         >>
>         >> Hi Gary,
>         >>
>         >> That's normal, this command has been removed since Dec'13, see
>         
> >>https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/pip
>         
> >>ermail/openstack-dev/2013-December/021649.html&k=oIvRg1%2BdGAgOoM1BIlLLqw
>         
> >>%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=Nf6%2Fy
>         
> >>HMSSchQO59xIcg6NKX1O2wlkkHHd42bYQim0k0%3D%0A&s=826274ca8ad9562b557322274a
>         >>cdacd97482338456820af8c330b76ea7639838
>         >>
>         
> >><https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/pi
>         
> >>permail/openstack-dev/2013-December/021649.html&k=oIvRg1%2BdGAgOoM1BIlLLq
>         
> >>w%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=B8yQ75
>         
> >>uRFa7dyD%2FbgPgO%2FMT2x229MkK1vHWBVAzpFtM%3D%0A&s=d9cbcbde99b7c88c15ca138
>         >>499de8f36edd4247ce351e41b241d675b09b79956>
>         >>
>         >> -Sylvain
>         >>
>         >>
>         >> 2014-02-17 13:00 GMT+01:00 Gary Kotton <gkot...@vmware.com
>         <mailto:gkot...@vmware.com>
>         >><mailto:gkot...@vmware.com <mailto:gkot...@vmware.com>>>:
>         >>
>         >>     Hi,
>         >>     It seems that the command 'reverify bug <number>' is not
>         working.
>         >>Anyone else experienced this lately.
>         >>     Thanks
>         >>     Gary
>         >>
>         >>     _______________________________________________
>         >>     OpenStack-dev mailing list
>         >>     OpenStack-dev@lists.openstack.org
>         <mailto:OpenStack-dev@lists.openstack.org>
>         >><mailto:OpenStack-dev@lists.openstack.org
>         <mailto:OpenStack-dev@lists.openstack.org>>
>         >>
>         
> >>https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi
>         
> >>-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r
>         
> >>=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=Nf6%2FyHMSSchQO59x
>         
> >>Icg6NKX1O2wlkkHHd42bYQim0k0%3D%0A&s=1a2a59024e18b786b5c2c13535b7f3d050363
>         >>ff628dc96b8561e12489d30c0bd
>         >>
>         
> >><https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cg
>         
> >>i-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&
>         
> >>r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=B8yQ75uRFa7dyD%2F
>         
> >>bgPgO%2FMT2x229MkK1vHWBVAzpFtM%3D%0A&s=21e7f4ebc24f0a13780981720f9332111d
>         >>d603f3c0661229dc90d82bfb5c3122>
>         >>
>         >>
>         >>
>         >>
>         >> _______________________________________________
>         >> OpenStack-dev mailing list
>         >> OpenStack-dev@lists.openstack.org
>         <mailto:OpenStack-dev@lists.openstack.org>
>         >>
>         
> >>https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi
>         
> >>-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r
>         
> >>=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=Nf6%2FyHMSSchQO59x
>         
> >>Icg6NKX1O2wlkkHHd42bYQim0k0%3D%0A&s=1a2a59024e18b786b5c2c13535b7f3d050363
>         >>ff628dc96b8561e12489d30c0bd
>         >>
>         >_______________________________________________
>         >OpenStack-dev mailing list
>         >OpenStack-dev@lists.openstack.org
>         <mailto:OpenStack-dev@lists.openstack.org>
>         
> >https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-
>         
> >bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=e
>         
> >H0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=Nf6%2FyHMSSchQO59xIcg
>         
> >6NKX1O2wlkkHHd42bYQim0k0%3D%0A&s=1a2a59024e18b786b5c2c13535b7f3d050363ff62
>         >8dc96b8561e12489d30c0bd
> 
> 
>         _______________________________________________
>         OpenStack-dev mailing list
>         OpenStack-dev@lists.openstack.org
>         <mailto:OpenStack-dev@lists.openstack.org>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
>     _______________________________________________
>     OpenStack-dev mailing list
>     OpenStack-dev@lists.openstack.org
>     <mailto:OpenStack-dev@lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> -- 
> Sincerely yours,
> Sergey Lukjanov
> Savanna Technical Lead
> Mirantis Inc.
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to