Re: [openstack-dev] [all][infra] Zuul v3 migration update

[Doug Hellmann]

Excerpts from Clark Boylan's message of 2017-09-28 18:40:41 -0700:
> On Wed, Sep 27, 2017, at 03:24 PM, Monty Taylor wrote:
> > Hey everybody,
> > 
> > We're there. It's ready.
> > 
> > We've worked through all of the migration script issues and are happy 
> > with the results. The cutover trigger is primed and ready to go.
> > 
> > But as it's 21:51 UTC / 16:52 US Central it's a short day to be 
> > available to respond to the questions folks may have... so we're going 
> > to postpone one more day.
> > 
> > Since it's all ready to go we'll be looking at flipping the switch first 
> > thing in the morning. (basically as soon as the West Coast wakes up and 
> > is ready to go)
> > 
> > The project-config repo should still be considered frozen except for 
> > migration-related changes. Hopefully we'll be able to flip the final 
> > switch early tomorrow.
> > 
> > If you haven't yet, please see [1] for information about the transition.
> > 
> > [1] https://docs.openstack.org/infra/manual/zuulv3.html
> > 
> 
> Its done! Except for all the work to make jobs run properly. Early today

Woo!

> (PDT) we converted everything over to our auto generated Zuulv3 config.
> Since then we've been working to address problems in job configs.
> 
> These problems include:
> Missing inclusion of the requirements repo for constraints in some
> jobs
> Configuration of python35 unittest jobs in some cases
> Use of sudo checking not working properly
> Multinode jobs not having multinode nodesets
> 
> Known issues we will continue to work on:
> Multinode devstack and grenade jobs are not working quite right
> Releasenote jobs not working due to use of origin/ refs in git

This problem should be fixed by the reno enhancement in
https://review.openstack.org/#/c/508324/, which I will release when I'm
given the all-clear for the release tagging jobs.

> It looks like we may not have job branch exclusions in place for all
> cases
> The zuul-cloner shim may not work in all cases. We are tracking down
> and fixing the broken corner cases.
> 
> Keep in mind that with things in flux, there is a good chance that
> changes enqueued to the gate will fail. It is a good idea to check
> recent check queue results before approving changes.
> 
> I don't think we've found any deal breaker problems at this point. I am
> sure there are many more than I have listed above. Please feel free to
> ask us about any errors. For the adventurous, fixing problems is likely
> a great way to get familiar with the new system. You'll want to start by
> fixing errors in openstack-infra/openstack-zuul-jobs/playbooks/legacy.
> Once that stabilizes the next step is writing native job configs within
> your project tree. Documentation can be found at
> https://docs.openstack.org/infra/manual/zuulv3.html. I expect we'll
> spend the next few days ironing out the transition.
> 
> Thank you for your patience,
> Clark
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 migration update

[Luke Hinds]

On Fri, Sep 29, 2017 at 2:40 AM, Clark Boylan  wrote:

> On Wed, Sep 27, 2017, at 03:24 PM, Monty Taylor wrote:
> > Hey everybody,
> >
> > We're there. It's ready.
> >
> > We've worked through all of the migration script issues and are happy
> > with the results. The cutover trigger is primed and ready to go.
> >
> > But as it's 21:51 UTC / 16:52 US Central it's a short day to be
> > available to respond to the questions folks may have... so we're going
> > to postpone one more day.
> >
> > Since it's all ready to go we'll be looking at flipping the switch first
> > thing in the morning. (basically as soon as the West Coast wakes up and
> > is ready to go)
> >
> > The project-config repo should still be considered frozen except for
> > migration-related changes. Hopefully we'll be able to flip the final
> > switch early tomorrow.
> >
> > If you haven't yet, please see [1] for information about the transition.
> >
> > [1] https://docs.openstack.org/infra/manual/zuulv3.html
> >
>
> Its done! Except for all the work to make jobs run properly. Early today
> (PDT) we converted everything over to our auto generated Zuulv3 config.
> Since then we've been working to address problems in job configs.
>
> These problems include:
> Missing inclusion of the requirements repo for constraints in some
> jobs
> Configuration of python35 unittest jobs in some cases
> Use of sudo checking not working properly
> Multinode jobs not having multinode nodesets
>
> Known issues we will continue to work on:
> Multinode devstack and grenade jobs are not working quite right
> Releasenote jobs not working due to use of origin/ refs in git
> It looks like we may not have job branch exclusions in place for all
> cases
> The zuul-cloner shim may not work in all cases. We are tracking down
> and fixing the broken corner cases.
>
> Keep in mind that with things in flux, there is a good chance that
> changes enqueued to the gate will fail. It is a good idea to check
> recent check queue results before approving changes.
>
> I don't think we've found any deal breaker problems at this point. I am
> sure there are many more than I have listed above. Please feel free to
> ask us about any errors. For the adventurous, fixing problems is likely
> a great way to get familiar with the new system. You'll want to start by
> fixing errors in openstack-infra/openstack-zuul-jobs/playbooks/legacy.
> Once that stabilizes the next step is writing native job configs within
> your project tree. Documentation can be found at
> https://docs.openstack.org/infra/manual/zuulv3.html. I expect we'll
> spend the next few days ironing out the transition.
>
> Thank you for your patience,
> Clark
>
>
Hi,

We have a couple of failures in Bandit's gvate I wanted to flag up (fungi
believes it might be from missing openstack/requirements as a required repo
in the job definition):

Glance Store:

http://logs.openstack.org/44/504544/6/check/legacy-bandit-integration-glance_store/900f88a/job-output.txt.gz


Magnum client:

http://logs.openstack.org/44/504544/6/check/legacy-bandit-integration-python-magnumclient/ba50a54/job-output.txt.gz

Thanks, and do let me know if I need to do anything on Bandits side.

Luke
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 migration update

[Clark Boylan]

On Wed, Sep 27, 2017, at 03:24 PM, Monty Taylor wrote:
> Hey everybody,
> 
> We're there. It's ready.
> 
> We've worked through all of the migration script issues and are happy 
> with the results. The cutover trigger is primed and ready to go.
> 
> But as it's 21:51 UTC / 16:52 US Central it's a short day to be 
> available to respond to the questions folks may have... so we're going 
> to postpone one more day.
> 
> Since it's all ready to go we'll be looking at flipping the switch first 
> thing in the morning. (basically as soon as the West Coast wakes up and 
> is ready to go)
> 
> The project-config repo should still be considered frozen except for 
> migration-related changes. Hopefully we'll be able to flip the final 
> switch early tomorrow.
> 
> If you haven't yet, please see [1] for information about the transition.
> 
> [1] https://docs.openstack.org/infra/manual/zuulv3.html
> 

Its done! Except for all the work to make jobs run properly. Early today
(PDT) we converted everything over to our auto generated Zuulv3 config.
Since then we've been working to address problems in job configs.

These problems include:
Missing inclusion of the requirements repo for constraints in some
jobs
Configuration of python35 unittest jobs in some cases
Use of sudo checking not working properly
Multinode jobs not having multinode nodesets

Known issues we will continue to work on:
Multinode devstack and grenade jobs are not working quite right
Releasenote jobs not working due to use of origin/ refs in git
It looks like we may not have job branch exclusions in place for all
cases
The zuul-cloner shim may not work in all cases. We are tracking down
and fixing the broken corner cases.

Keep in mind that with things in flux, there is a good chance that
changes enqueued to the gate will fail. It is a good idea to check
recent check queue results before approving changes.

I don't think we've found any deal breaker problems at this point. I am
sure there are many more than I have listed above. Please feel free to
ask us about any errors. For the adventurous, fixing problems is likely
a great way to get familiar with the new system. You'll want to start by
fixing errors in openstack-infra/openstack-zuul-jobs/playbooks/legacy.
Once that stabilizes the next step is writing native job configs within
your project tree. Documentation can be found at
https://docs.openstack.org/infra/manual/zuulv3.html. I expect we'll
spend the next few days ironing out the transition.

Thank you for your patience,
Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][infra] Zuul v3 migration update

[Monty Taylor]

Hey everybody,

We're there. It's ready.

We've worked through all of the migration script issues and are happy 
with the results. The cutover trigger is primed and ready to go.


But as it's 21:51 UTC / 16:52 US Central it's a short day to be 
available to respond to the questions folks may have... so we're going 
to postpone one more day.


Since it's all ready to go we'll be looking at flipping the switch first 
thing in the morning. (basically as soon as the West Coast wakes up and 
is ready to go)


The project-config repo should still be considered frozen except for 
migration-related changes. Hopefully we'll be able to flip the final 
switch early tomorrow.


If you haven't yet, please see [1] for information about the transition.

[1] https://docs.openstack.org/infra/manual/zuulv3.html

Thanks,

Monty

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 migration update

[Jay Pipes]

On 09/27/2017 03:49 AM, Flavio Percoco wrote:
Just wanted to say thanks to all of you for the hard work. I can only 
imagine

how hard it must be to do this migration without causing downtimes.


+1000

Thank you so much for the hard work the infra team has put into making 
this migration as painless for the community as possible. Your efforts 
have certainly not gone unnoticed.


All the best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 migration update

[Flavio Percoco]

Just wanted to say thanks to all of you for the hard work. I can only imagine
how hard it must be to do this migration without causing downtimes.

Flavio

On 26/09/17 18:04 -0500, Monty Taylor wrote:

Hey everybody,

We got significantly further along with our Zuul v3 rollout today. We
uncovered some fun bugs in the migration but were able to fix most of
them rather quickly.

We've pretty much run out of daylight though for the majority of the
team and there is a tricky zuul-cloner related issue to deal with, so
we're not going to push things further tonight. We're leaving most of
today's work in place, having gotten far enough that we feel
comfortable not rolling back.

The project-config repo should still be considered frozen except for
migration-related changes. Hopefully we'll be able to flip the final
switch early tomorrow.

If you haven't yet, please see [1] for information about the transition.

[1] https://docs.openstack.org/infra/manual/zuulv3.html

Thanks,

Monty

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][infra] Zuul v3 migration update

[Monty Taylor]

Hey everybody,

We got significantly further along with our Zuul v3 rollout today. We 
uncovered some fun bugs in the migration but were able to fix most of 
them rather quickly.


We've pretty much run out of daylight though for the majority of the 
team and there is a tricky zuul-cloner related issue to deal with, so 
we're not going to push things further tonight. We're leaving most of 
today's work in place, having gotten far enough that we feel comfortable 
not rolling back.


The project-config repo should still be considered frozen except for 
migration-related changes. Hopefully we'll be able to flip the final 
switch early tomorrow.


If you haven't yet, please see [1] for information about the transition.

[1] https://docs.openstack.org/infra/manual/zuulv3.html

Thanks,

Monty

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][infra] Zuul v3 migration update

[James E. Blair]

Hi,

We started to migrate to Zuul v3 today, but ran into some problems which
delayed us sufficiently that we're going to suspend activity until
tomorrow.

In the mean time, please consider the project-config repository frozen
to all changes except those which are necessary for the migration.  We
should be able to lift this freeze as soon as we finish the migration.

If you haven't yet, please see [1] for information about the transition.

[1] https://docs.openstack.org/infra/manual/zuulv3.html

Thanks,

Jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev