Re: [openstack-dev] [all] Zuul v3 Status - and Rollback Information

[Mike Perez]

On 19:38 Oct 03, Jean-Philippe Evrard wrote:
> On Tue, Oct 3, 2017 at 5:40 PM, Monty Taylor  wrote:
> > Hey everybody!


 
> Hello,
> 
> I'd like to first thank everyone involved, doing this hard work.

Yes thank you all very much for your hardwork. I think we can all agree
"computers happen."

-- 
Mike Perez


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Zuul v3 Status - and Rollback Information

[Jean-Philippe Evrard]

On Tue, Oct 3, 2017 at 5:40 PM, Monty Taylor  wrote:
> Hey everybody!
>
> Thank you all for your patience over the last week as we've worked through
> the v3 rollout.  While we anticipated some problems, we've run into a number
> of unexpected issues, some of which will take some time to resolve.  It has
> become unreasonable to continue in our current state while they are being
> worked.
>
> With that in mind, we are going to perform a partial rollback to Zuul v2
> while we work on it so that teams can get work done. The details of that are
> as follows:
>
> The project-config repo remains frozen.  Generally we don't want to make
> changes to v2 jobs.  If a change must be made, it will need to be made to
> both v2 and v3 versions.  We will not run the migration script again.
>
> Zuul v3 will continue to run check and periodic jobs on all repos.  It will
> leave review messages, including +1/-1 votes.
>
> Our nodepool quota will be allocated 80% to Zuul v2, and 20% to ZUul v3.
> This will slow v2 down slightly, but allow us to continue to exercise v3
> enough to find problems.
>
> Zuul v2 and v3 can not both gate a project or set of projects.  In general,
> Zuul v2 will be gating all projects, except the few projects that are
> specifically v3-only: zuul-jobs, openstack-zuul-jobs, project-config, and
> zuul itself.
>
> We appreciate that some projects would prefer to use v3 exclusively, even
> while we continue to work to stabilize it.  However, in order to complete
> our work as quickly as possible, we may need to restart frequently or take
> extended v3 downtimes.  Because of this, and the reduced capacity that v3
> will be running with, we will keep the set of projects gating under v3
> limited to only those necessary.  But keep in mind, v3 will still be running
> check jobs on all projects, so you can continue to iterate on v3 job content
> in check.
>
> If you modified a script in your repo that is called from a job to work in
> v3, you may need to modify it to be compatible with both.  If you need to
> determine whether you are running under Zuul v2 or under v3 with legacy
> compatibility shims, check for the LOG_PATH environment variable.  It will
> only be present when running under Zuul v2 (and it is the variable that we
> are least likely to add to the v3 compatibility shim).
>
> Again - thank you all for your patience, and for all of the great work
> people have done working through the issues we've uncovered. As soon as
> we've got a handle on the most critical issues, we'll plan another
> roll-forward ... hopefully in a week or two, but we'll send out status
> updates as we have them.
>
> Thanks!
> Monty
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Hello,

I'd like to first thank everyone involved, doing this hard work.

As an extra comment, please remember that Newton EOL is next week, so
it's maybe worth waiting after
that for the next full rollout/roll-forward (whatever the term is!) to
avoid the overlap of critical events in the same timeframe.

Best regards,
Jean-Philippe Evrard (evrardjp)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Zuul v3 Status - and Rollback Information

[Sean Dague]

On 10/03/2017 01:06 PM, Jeremy Stanley wrote:
> On 2017-10-03 12:00:44 -0500 (-0500), Matt Riedemann wrote:
>> On 10/3/2017 11:40 AM, Monty Taylor wrote:
>>> Our nodepool quota will be allocated 80% to Zuul v2, and 20% to ZUul v3. 
>>> This will slow v2 down slightly, but allow us to continue to exercise v3
>>> enough to find problems.
>>>
>>> Zuul v2 and v3 can not both gate a project or set of projects.  In
>>> general, Zuul v2 will be gating all projects, except the few projects that
>>> are specifically v3-only: zuul-jobs, openstack-zuul-jobs, project-config,
>>> and zuul itself.
>>
>> So if v3 is in check and periodic, and will be restarted and offline at
>> times, doesn't that mean we could have patches waiting for an extended
>> period of time on v3 results when the v2 jobs are long done? Or do the v3
>> jobs just timeout and being non-voting shouldn't impact the overall score?
> 
> Not at all, since v2 will also be running check jobs itself and it
> only pays attention to its own results for that puropose. So
> basically while we're rolled back to v2 you'll see both "Jenkins"
> (v2 account) and "Zuul" (v3 account) reporting check results most of
> the time but you can consider the v3 results merely an advisory
> indicator of whether your v3 jobs are working correctly.

Right, so zuul v3 will effectively be treated like 3rd party CI during
the transition. Sounds good.

The 80 / 20 split seems very reasonable, and a good way to let teams get
back to work while letting the v3 effort make forward progress with real
load to smoke out the issues.

Thanks for flipping over to this model.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Zuul v3 Status - and Rollback Information

[Jeremy Stanley]

On 2017-10-03 12:00:44 -0500 (-0500), Matt Riedemann wrote:
> On 10/3/2017 11:40 AM, Monty Taylor wrote:
> >Our nodepool quota will be allocated 80% to Zuul v2, and 20% to ZUul v3. 
> >This will slow v2 down slightly, but allow us to continue to exercise v3
> >enough to find problems.
> >
> >Zuul v2 and v3 can not both gate a project or set of projects.  In
> >general, Zuul v2 will be gating all projects, except the few projects that
> >are specifically v3-only: zuul-jobs, openstack-zuul-jobs, project-config,
> >and zuul itself.
> 
> So if v3 is in check and periodic, and will be restarted and offline at
> times, doesn't that mean we could have patches waiting for an extended
> period of time on v3 results when the v2 jobs are long done? Or do the v3
> jobs just timeout and being non-voting shouldn't impact the overall score?

Not at all, since v2 will also be running check jobs itself and it
only pays attention to its own results for that puropose. So
basically while we're rolled back to v2 you'll see both "Jenkins"
(v2 account) and "Zuul" (v3 account) reporting check results most of
the time but you can consider the v3 results merely an advisory
indicator of whether your v3 jobs are working correctly.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Zuul v3 Status - and Rollback Information

[Clark Boylan]

On Tue, Oct 3, 2017, at 10:00 AM, Matt Riedemann wrote:
> On 10/3/2017 11:40 AM, Monty Taylor wrote:
> > Our nodepool quota will be allocated 80% to Zuul v2, and 20% to ZUul 
> > v3.  This will slow v2 down slightly, but allow us to continue to 
> > exercise v3 enough to find problems.
> > 
> > Zuul v2 and v3 can not both gate a project or set of projects.  In 
> > general, Zuul v2 will be gating all projects, except the few projects 
> > that are specifically v3-only: zuul-jobs, openstack-zuul-jobs, 
> > project-config, and zuul itself.
> 
> So if v3 is in check and periodic, and will be restarted and offline at 
> times, doesn't that mean we could have patches waiting for an extended 
> period of time on v3 results when the v2 jobs are long done? Or do the 
> v3 jobs just timeout and being non-voting shouldn't impact the overall 
> score?

v2 will do all of the gating and will for the most part completely
ignore what v3 is doing. The exception to this is any -2's that zuul has
left will need to be cleared out (fungi is working on this as I write
this email).

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Zuul v3 Status - and Rollback Information

[Matt Riedemann]

On 10/3/2017 11:40 AM, Monty Taylor wrote:
Our nodepool quota will be allocated 80% to Zuul v2, and 20% to ZUul 
v3.  This will slow v2 down slightly, but allow us to continue to 
exercise v3 enough to find problems.


Zuul v2 and v3 can not both gate a project or set of projects.  In 
general, Zuul v2 will be gating all projects, except the few projects 
that are specifically v3-only: zuul-jobs, openstack-zuul-jobs, 
project-config, and zuul itself.


So if v3 is in check and periodic, and will be restarted and offline at 
times, doesn't that mean we could have patches waiting for an extended 
period of time on v3 results when the v2 jobs are long done? Or do the 
v3 jobs just timeout and being non-voting shouldn't impact the overall 
score?


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Zuul v3 Status - and Rollback Information

[Monty Taylor]

Hey everybody!

Thank you all for your patience over the last week as we've worked 
through the v3 rollout.  While we anticipated some problems, we've run 
into a number of unexpected issues, some of which will take some time to 
resolve.  It has become unreasonable to continue in our current state 
while they are being worked.


With that in mind, we are going to perform a partial rollback to Zuul v2 
while we work on it so that teams can get work done. The details of that 
are as follows:


The project-config repo remains frozen.  Generally we don't want to make 
changes to v2 jobs.  If a change must be made, it will need to be made 
to both v2 and v3 versions.  We will not run the migration script again.


Zuul v3 will continue to run check and periodic jobs on all repos.  It 
will leave review messages, including +1/-1 votes.


Our nodepool quota will be allocated 80% to Zuul v2, and 20% to ZUul 
v3.  This will slow v2 down slightly, but allow us to continue to 
exercise v3 enough to find problems.


Zuul v2 and v3 can not both gate a project or set of projects.  In 
general, Zuul v2 will be gating all projects, except the few projects 
that are specifically v3-only: zuul-jobs, openstack-zuul-jobs, 
project-config, and zuul itself.


We appreciate that some projects would prefer to use v3 exclusively, 
even while we continue to work to stabilize it.  However, in order to 
complete our work as quickly as possible, we may need to restart 
frequently or take extended v3 downtimes.  Because of this, and the 
reduced capacity that v3 will be running with, we will keep the set of 
projects gating under v3 limited to only those necessary.  But keep in 
mind, v3 will still be running check jobs on all projects, so you can 
continue to iterate on v3 job content in check.


If you modified a script in your repo that is called from a job to work 
in v3, you may need to modify it to be compatible with both.  If you 
need to determine whether you are running under Zuul v2 or under v3 with 
legacy compatibility shims, check for the LOG_PATH environment 
variable.  It will only be present when running under Zuul v2 (and it is 
the variable that we are least likely to add to the v3 compatibility shim).


Again - thank you all for your patience, and for all of the great work 
people have done working through the issues we've uncovered. As soon as 
we've got a handle on the most critical issues, we'll plan another 
roll-forward ... hopefully in a week or two, but we'll send out status 
updates as we have them.


Thanks!
Monty

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev