Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel returns

[Jeremy Stanley]

There is no Jenkins, only Zuul.

Zuul v3 rollout is complete as of 23:00 UTC. At that time, v2 jobs
reported by the "Jenkins" account ceased to be relevant, and v3 jobs
reported by the "Zuul" account are now used to determine whether
your changes can merge.

We have an expedited priority "infra-check" pipeline for changes to
the project-config repository so that emergency fixes for legacy
jobs can be merged quickly, and we'll be keeping this for some time
after the transition until it ceases to be necessary. If you're
still seeing any unexpected failures from migrated "legacy" jobs for
your projects and aren't immediately certain how to address these,
please bring them to our attention as soon as possible (in
#openstack-infra on the Freenode IRC network, or replying on-list to
this announcement) and we're happy to help you work on fixing them.
It also helps to get them onto the triage list we have been
maintaining here:

https://etherpad.openstack.org/p/zuulv3-issues

We're assembling a sort of FAQ in the migration guide:

https://docs.openstack.org/infra/manual/zuulv3.html

...and we also have some more content in progress at:

https://etherpad.openstack.org/p/zuulv3-migration-faq

Thanks for your patience throughout this process, and hopefully
you'll agree the new features were worth the wait!
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel returns

[Sam Matzek]

Thanks.  I will try to fix the legacy-trove-functional.

On Sat, Oct 14, 2017 at 12:58 AM, Andreas Jaeger  wrote:
> On 2017-10-14 03:30, Sam Matzek wrote:
>> The legacy-trove-functional-dsvm-mysql and
>> legacy-trove-legacy-functional-dsvm-mysql jobs are running the wrong
>> post_test_hook and have the trove-integration project in $PROJECTS.
>
> Looking at zuul/layout.yaml the change
> trove-legacy-functional-dsvm-mysql should only run on stable/newton -
> but trove-functional-dsvm-mysql on anything newer.
> done with https://review.openstack.org/511997 Fix trove
> legacy-trove-legacy-functional-dsvm-mysql
>
>> As such they will always vote -1. The functional integration tests
>> moved into Trove proper in Ocata.  I've added more details and links
>> to the zuulv3-issues etherpad.
>>
>> I'm not familiar enough with the job definitions to be able to work on
>> the fix reviews for these but would like to learn.
>
> Those are in openstack-zuul-jobs, the file is
> http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/trove-functional-dsvm-mysql/run.yaml.
>
> Do you want to fix it yourself and sync with the v2 version that is in
> project-config/jenkins/jobs/trove.yaml?
>
> Andreas
> --
>  Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
>   SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
>GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>HRB 21284 (AG Nürnberg)
> GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel returns

[Andreas Jaeger]

On 2017-10-14 03:30, Sam Matzek wrote:
> The legacy-trove-functional-dsvm-mysql and
> legacy-trove-legacy-functional-dsvm-mysql jobs are running the wrong
> post_test_hook and have the trove-integration project in $PROJECTS.

Looking at zuul/layout.yaml the change
trove-legacy-functional-dsvm-mysql should only run on stable/newton -
but trove-functional-dsvm-mysql on anything newer.
done with https://review.openstack.org/511997 Fix trove
legacy-trove-legacy-functional-dsvm-mysql

> As such they will always vote -1. The functional integration tests
> moved into Trove proper in Ocata.  I've added more details and links
> to the zuulv3-issues etherpad.
> 
> I'm not familiar enough with the job definitions to be able to work on
> the fix reviews for these but would like to learn.

Those are in openstack-zuul-jobs, the file is
http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/trove-functional-dsvm-mysql/run.yaml.

Do you want to fix it yourself and sync with the v2 version that is in
project-config/jenkins/jobs/trove.yaml?

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel returns

[Sam Matzek]

The legacy-trove-functional-dsvm-mysql and
legacy-trove-legacy-functional-dsvm-mysql jobs are running the wrong
post_test_hook and have the trove-integration project in $PROJECTS.
As such they will always vote -1. The functional integration tests
moved into Trove proper in Ocata.  I've added more details and links
to the zuulv3-issues etherpad.

I'm not familiar enough with the job definitions to be able to work on
the fix reviews for these but would like to learn.

Sam Matzek

On Fri, Oct 13, 2017 at 2:57 PM, Jeremy Stanley  wrote:
> The tl;dr is that we're planning to roll forward out of our partial
> Zuul v3 rollback starting at 22:00 UTC on Sunday October 15 (this
> weekend), so expect some CI downtime and all of the benefits (though
> hopefully none of the drawbacks!) you witnessed when we tried the
> first time. At that time, v2 jobs reported by the "Jenkins" account
> will cease to be relevant, and v3 jobs reported by the "Zuul"
> account will be used to determine whether your changes can merge.
>
> As Monty noted earlier in the week, our plans to roll forward onto
> Zuul v3 were halted by a number of unrelated infrastructure fires
> which demanded our immediate attention as a team, so we reluctantly
> postponed. Following a (brief) hiatus yesterday, check pipeline
> processing has been readded to zuulv3.openstack.org for all projects
> so you can get fresh results back on v3 jobs between now and our
> maintenance if you happen to be around testing it out. We also still
> have an expedited priority "infra-check" pipeline on zuulv3 for
> changes to the project-config repository so that emergency fixes
> for legacy jobs can be merged quickly, and we'll be keeping this for
> some time after the transition until it ceases to be necessary.
>
> We anticipate an outage for the CI system of somewhere between 30-60
> minutes starting at 22:00 UTC on Sunday October 15. In the meantime,
> if you've been digging into recent legacy job failures for your
> projects you should consider trying to bring them to our attention
> as soon as possible (in #openstack-infra on the Freenode IRC
> network, or replying on-list to this announcement) and work on
> fixing them if you're familiar enough with the failures to do so
> yourself. It also helps to get them onto the triage list we have
> been maintaining here:
>
> https://etherpad.openstack.org/p/zuulv3-issues
>
> We're assembling a sort of FAQ in the migration guide
> here:
>
> https://docs.openstack.org/infra/manual/zuulv3.html
>
> ...and we also have some more content in progress at:
>
> https://etherpad.openstack.org/p/zuulv3-migration-faq
>
> Here's to the week ahead!
> --
> Jeremy Stanley
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel

[Jeremy Stanley]

On 2017-10-13 07:15:03 -0600 (-0600), Boden Russell wrote:
> How can projects validate zuul v3 jobs in our current state to prepare
> for the transition?
> Some projects don't even have a verified zuul v3 patch [1] and thus
> really have no way to test and work through v3 issues (IIUC).
> 
> Is there a way we can leave v3 non-gating and let projects request
> moving to gating v3 jobs on a per-project basis (e.g. project X is ready
> for v3, make v3 gating for X now)? This would allow them the time they
> need to transition with minimal impact to their pipeline.
[...]

Yes, now that we have the situation with the logs site under
control, we're working on getting v3 running check jobs for all
projects again in an advisory fashion (hopefully within the next few
hours).
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel

[Boden Russell]

On 10/10/17 3:40 AM, Andreas Jaeger wrote:
> The common jobs have been fixed whenever bugs got reported. So, if you
> have current failures, tell us. 

How can projects validate zuul v3 jobs in our current state to prepare
for the transition?
Some projects don't even have a verified zuul v3 patch [1] and thus
really have no way to test and work through v3 issues (IIUC).

Is there a way we can leave v3 non-gating and let projects request
moving to gating v3 jobs on a per-project basis (e.g. project X is ready
for v3, make v3 gating for X now)? This would allow them the time they
need to transition with minimal impact to their pipeline.


[1]
https://review.openstack.org/#/q/project:openstack/vmware-nsx+label:Verified%252Czuul

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel

[James E. Blair]

Gary Kotton  writes:

> Hi,
> At the moment the neutron, neutron-lib and many of the decomposed projects 
> are still failing with the v3. Does this mean that we are broken from the 
> 11th?
> For the decomposed projects we have a work around to help address this in the 
> short term – need to increase timeout and need a flag from Zuul3 that is not 
> part of Jenkins - 
> https://github.com/openstack/vmware-nsx/blob/master/tools/tox_install_project.sh#L37
>  (can we have ZUUL3_CLONER?)
> Thanks
> Gary

In that script $ZUUL_CLONER points to /usr/zuul-env/bin/zuul-cloner
which will exist for auto-converted legacy jobs in v3.  If such a job is
not working, let's take a look at a log and dig into it.

That will *not* be present for non-legacy jobs.  I think once the dust
settles, we can work on using some much nicer facilities that v3
provides for doing that sort of thing in new native v3 jobs.

-Jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel

[Sean McGinnis]

> 
> The common jobs have been fixed whenever bugs got reported. So, if you
> have current failures, tell us. Let me readd a quote from Jeremy:
> 
> >> In
> >> the meantime, if you've been digging into recent legacy job failures
> >> for your projects you should consider trying to bring them to our
> >> attention as soon as possible (in #openstack-infra on the Freenode
> >> IRC network, or replying on-list to this announcement) and work on
> >> fixing them if you're familiar enough with the failures to do so
> >> yourself.
> 

Well, there's a difference between "digging into recent legacy job failures"
and just waiting for all the red to go away. I have not dug into any failures
since it seemed like things were in a state of flux and therefore not worth
spending the time yet. But I guess now is the time.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel

[Gary Kotton]

Hi,
At the moment the neutron, neutron-lib and many of the decomposed projects are 
still failing with the v3. Does this mean that we are broken from the 11th?
For the decomposed projects we have a work around to help address this in the 
short term – need to increase timeout and need a flag from Zuul3 that is not 
part of Jenkins - 
https://github.com/openstack/vmware-nsx/blob/master/tools/tox_install_project.sh#L37
 (can we have ZUUL3_CLONER?)
Thanks
Gary

On 10/10/17, 2:59 AM, "Jeremy Stanley"  wrote:

The tl;dr is that we're planning to roll forward out of our partial
Zuul v3 rollback starting at 11:00 UTC on Wednesday October 11
(a little over 35 hours from now), so expect some CI downtime and
all of the benefits (though hopefully none of the drawbacks!) you
witnessed when we tried the first time.

It's been right at a week since we instituted a partial rollback of
our initial v3 roll-out. That week has been filled by diagnosing and
fixing all of the misbehaviors and performance degradation we
identified, including some new issues we discovered while running
under even heavier load and memory pressure artificially induced by
trying to fire check jobs for everything v2 was running but with
only a fraction of the node capacity. Numerous issues within the
translated legacy job configs were also fixed, and a bunch more of
them replaced by v3-native jobs.

We anticipate an outage for the CI system of somewhere between 30-60
minutes starting at 11:00 UTC on Wednesday October 11. Once
complete, we'll send a follow-up announcement along with a link to
where we're coordinating and triaging any newly observed issues. In
the meantime, if you've been digging into recent legacy job failures
for your projects you should consider trying to bring them to our
attention as soon as possible (in #openstack-infra on the Freenode
IRC network, or replying on-list to this announcement) and work on
fixing them if you're familiar enough with the failures to do so
yourself. We're assembling a sort of FAQ in the migration guide
here:

https://docs.openstack.org/infra/manual/zuulv3.html

...and we also have some more content in progress at:

https://etherpad.openstack.org/p/zuulv3-migration-faq

In summary, Zuul v3 is looking better than ever, and we hope you'll
be as pleased with it as we are!
-- 
Jeremy Stanley


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel

[Andreas Jaeger]

On 2017-10-10 11:01, Sean McGinnis wrote:
> On Mon, Oct 09, 2017 at 11:59:31PM +, Jeremy Stanley wrote:
>> The tl;dr is that we're planning to roll forward out of our partial
>> Zuul v3 rollback starting at 11:00 UTC on Wednesday October 11
>> (a little over 35 hours from now), so expect some CI downtime and
>> all of the benefits (though hopefully none of the drawbacks!) you
>> witnessed when we tried the first time.
>>
> 
> My perception has been there has still been a high occurrance of false 
> failures
> on the patches where zuul has still run, at least on the Cinder side. Are
> things expected to be peachy by this point?


The common jobs have been fixed whenever bugs got reported. So, if you
have current failures, tell us. Let me readd a quote from Jeremy:

>> In
>> the meantime, if you've been digging into recent legacy job failures
>> for your projects you should consider trying to bring them to our
>> attention as soon as possible (in #openstack-infra on the Freenode
>> IRC network, or replying on-list to this announcement) and work on
>> fixing them if you're familiar enough with the failures to do so
>> yourself.

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra] Zuul v3 rollout, the sequel

[Sean McGinnis]

On Mon, Oct 09, 2017 at 11:59:31PM +, Jeremy Stanley wrote:
> The tl;dr is that we're planning to roll forward out of our partial
> Zuul v3 rollback starting at 11:00 UTC on Wednesday October 11
> (a little over 35 hours from now), so expect some CI downtime and
> all of the benefits (though hopefully none of the drawbacks!) you
> witnessed when we tried the first time.
> 

My perception has been there has still been a high occurrance of false failures
on the patches where zuul has still run, at least on the Cinder side. Are
things expected to be peachy by this point?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev