Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-10-05 Thread Tony Breeds 
On Thu, Oct 05, 2017 at 09:06:24PM +0900, Takashi Yamamoto wrote:
 
> i was told this is a fix. https://review.openstack.org/#/c/508898/

Yup the transition will be complete when
https://review.openstack.org/#/c/509855/ merges.  At that point we'll be
using all the zuulv3 goodness :)

Thanks again to Monty and the whole infra team who despite being extremely
busy with this work are just as helpful as always :D

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-10-05 Thread Takashi Yamamoto 
On Thu, Oct 5, 2017 at 8:58 PM, Pierre Riteau  wrote:
> On 29 Sep 2017, at 15:58, Monty Taylor  wrote:
>
> tl;dr - If you're having issues with your jobs, check the FAQ, this email
> and followups on this thread for mentions of them. If it's an issue with
> your job and you can spot it (bad config) just submit a patch with topic
> 'zuulv3'. If it's bigger/weirder/you don't know - we'd like to ask that you
> send a follow up email to this thread so that we can ensure we've got them
> all and so that others can see it too.
>
>
> Hello,
>
> Automated requirements updates [1] are all failing with the same error in
> the run.yaml playbook of the legacy-requirements job, e.g. in
> python-blazarclient [2].
> The command `/usr/local/jenkins/slave_scripts/project-requirements-change.py
> $ZUUL_BRANCH` fails with:
>
> SystemError: error: pathspec 'remotes/origin/master' did not match any
> file(s) known to git.
>
> or, for patches using the Pike stable branch:
>
> SystemError: error: pathspec 'remotes/origin/stable/pike' did not match
> any file(s) known to git.

i was told this is a fix. https://review.openstack.org/#/c/508898/

>
> [1]
> https://review.openstack.org/#/q/topic:openstack/requirements+status:open
> [2]
> http://logs.openstack.org/50/509450/9/check/legacy-requirements/914355a/ara/
>
> Best regards,
> Pierre Riteau
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-10-05 Thread Pierre Riteau 
> On 29 Sep 2017, at 15:58, Monty Taylor  wrote:
> 
> tl;dr - If you're having issues with your jobs, check the FAQ, this email and 
> followups on this thread for mentions of them. If it's an issue with your job 
> and you can spot it (bad config) just submit a patch with topic 'zuulv3'. If 
> it's bigger/weirder/you don't know - we'd like to ask that you send a follow 
> up email to this thread so that we can ensure we've got them all and so that 
> others can see it too.

Hello,

Automated requirements updates [1] are all failing with the same error in the 
run.yaml playbook of the legacy-requirements job, e.g. in python-blazarclient 
[2].
The command `/usr/local/jenkins/slave_scripts/project-requirements-change.py 
$ZUUL_BRANCH` fails with:

SystemError: error: pathspec 'remotes/origin/master' did not match any 
file(s) known to git.

or, for patches using the Pike stable branch:

SystemError: error: pathspec 'remotes/origin/stable/pike' did not match any 
file(s) known to git.

[1] https://review.openstack.org/#/q/topic:openstack/requirements+status:open 

[2] 
http://logs.openstack.org/50/509450/9/check/legacy-requirements/914355a/ara/ 


Best regards,
Pierre Riteau

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-10-03 Thread Dan Smith 
> Any update on where we stand on issues now? Because every single patch I
> tried to land yesterday was killed by POST_FAILURE in various ways.
> Including some really small stuff - https://review.openstack.org/#/c/324720/

Yeah, Nova has only landed eight patches since Thursday. Most of those are 
test-only patches that run a subset of jobs, and a couple that landed in the 
wee hours when overall system load was low.

> Do we have a defined point on the calendar for getting the false
> negatives back below the noise threshold otherwise a rollback is
> implemented so that some of these issues can be addressed in parallel
> without holding up community development?

On Friday I was supportive of the decision to keep steaming forward instead of 
rolling back. Today, I’m a bit more concerned about light at the end of the 
tunnel. The infra folks have been hitting this hard for a long time, and for 
that I’m very appreciative. I too hope that we’re going to revisit mitigation 
strategies as we approach the weekiversary of being stuck.

-—Dan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-10-03 Thread Gary Kotton 
We have patches stuck for hours – only info is:
http://zuulv3.openstack.org/static/stream.html?uuid=128746a70c1843d7a94e887120ba381c=console.log
At the moment we are unable to do anything

On 10/3/17, 3:36 PM, "Boden Russell"  wrote:

On 10/3/17 5:17 AM, Sean Dague wrote:
> 
> Do we have a defined point on the calendar for getting the false
> negatives back below the noise threshold otherwise a rollback is
> implemented so that some of these issues can be addressed in parallel
> without holding up community development?

Along the same lines; where is the best place to get help with zuul v3
issues? The neutron-lib gate is on the floor with multiple problems; 2
broken gating jobs preventing patches from landing and all periodic jobs
broken preventing (safe) releases of neutron-lib. I've been adding the
issues to the etherpad [1] and trying to work through them solo, but
progress is very slow.


[1] https://etherpad.openstack.org/p/zuulv3-migration-faq

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-10-03 Thread Boden Russell 
On 10/3/17 5:17 AM, Sean Dague wrote:
> 
> Do we have a defined point on the calendar for getting the false
> negatives back below the noise threshold otherwise a rollback is
> implemented so that some of these issues can be addressed in parallel
> without holding up community development?

Along the same lines; where is the best place to get help with zuul v3
issues? The neutron-lib gate is on the floor with multiple problems; 2
broken gating jobs preventing patches from landing and all periodic jobs
broken preventing (safe) releases of neutron-lib. I've been adding the
issues to the etherpad [1] and trying to work through them solo, but
progress is very slow.


[1] https://etherpad.openstack.org/p/zuulv3-migration-faq

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-10-03 Thread Sean Dague 
Any update on where we stand on issues now? Because every single patch I
tried to land yesterday was killed by POST_FAILURE in various ways.
Including some really small stuff - https://review.openstack.org/#/c/324720/

That also includes the patch I'm told fixes some issues with zuul v3 in
the base devstack jobs - https://review.openstack.org/#/c/508344/3

It also appears that many of the skips stopped being a thing -
https://review.openstack.org/#/c/507527/ got a Tempest test run
attempted on it (though everything ended in Node failure).

Do we have a defined point on the calendar for getting the false
negatives back below the noise threshold otherwise a rollback is
implemented so that some of these issues can be addressed in parallel
without holding up community development?

-Sean

On 09/29/2017 10:58 AM, Monty Taylor wrote:
> Hey everybody!
> 
> tl;dr - If you're having issues with your jobs, check the FAQ, this
> email and followups on this thread for mentions of them. If it's an
> issue with your job and you can spot it (bad config) just submit a patch
> with topic 'zuulv3'. If it's bigger/weirder/you don't know - we'd like
> to ask that you send a follow up email to this thread so that we can
> ensure we've got them all and so that others can see it too.
> 
> ** Zuul v3 Migration Status **
> 
> If you haven't noticed the Zuul v3 migration - awesome, that means it's
> working perfectly for you.
> 
> If you have - sorry for the disruption. It turns out we have a REALLY
> complicated array of job content you've all created. Hopefully the pain
> of the moment will be offset by the ability for you to all take direct
> ownership of your awesome content... so bear with us, your patience is
> appreciated.
> 
> If you find yourself with some extra time on your hands while you wait
> on something, you may find it helpful to read:
> 
>   https://docs.openstack.org/infra/manual/zuulv3.html
> 
> We're adding content to it as issues arise. Unfortunately, one of the
> issues is that the infra manual publication job stopped working.
> 
> While the infra manual publication is being fixed, we're collecting FAQ
> content for it in an etherpad:
> 
>   https://etherpad.openstack.org/p/zuulv3-migration-faq
> 
> If you have a job issue, check it first to see if we've got an entry for
> it. Once manual publication is fixed, we'll update the etherpad to point
> to the FAQ section of the manual.
> 
> ** Global Issues **
> 
> There are a number of outstanding issues that are being worked. As of
> right now, there are a few major/systemic ones that we're looking in to
> that are worth noting:
> 
> * Zuul Stalls
> 
> If you say to yourself "zuul doesn't seem to be doing anything, did I do
> something wrong?", we're having an issue that jeblair and Shrews are
> currently tracking down with intermittent connection issues in the
> backend plumbing.
> 
> When it happens it's an across the board issue, so fixing it is our
> number one priority.
> 
> * Incorrect node type
> 
> We've got reports of things running on trusty that should be running on
> xenial. The job definitions look correct, so this is also under
> investigation.
> 
> * Multinode jobs having POST FAILURE
> 
> There is a bug in the log collection trying to collect from all nodes
> while the old jobs were designed to only collect from the 'primary'.
> Patches are up to fix this and should be fixed soon.
> 
> * Branch Exclusions being ignored
> 
> This has been reported and its cause is currently unknown.
> 
> Thank you all again for your patience! This is a giant rollout with a
> bunch of changes in it, so we really do appreciate everyone's
> understanding as we work through it all.
> 
> Monty
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-10-02 Thread Renat Akhmerov 
On 2 Oct 2017, 21:02 +0700, wrote:

>
> * Zuul Stalls
>
> If you say to yourself "zuul doesn't seem to be doing anything, did I do
> something wrong?", we're having an issue that jeblair and Shrews are
> currently tracking down with intermittent connection issues in the
> backend plumbing.

Hi Monty, does it make sense to recheck patches in this case?

Thanks

Renat Akhmerov
@Nokia

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-10-02 Thread Vega Cai 
Hi Mohammed,

Thanks for your suggestion. I have submitted a patch [1] to try to fix the
job configuration, and used [2] that depends on it to test whether the fix
works.

[1] https://review.openstack.org/#/c/508824/
[2] https://review.openstack.org/#/c/508496/

On Sat, 30 Sep 2017 at 20:31 Mohammed Naser  wrote:

> Hi Vega,
>
> Please check the document. Some jobs were migrated with incorrect nodesets
> and have to be switched to multinode in the job definition in
> openstack-zuul-jobs
>
> Good luck
> Mohammed
>
> Sent from my iPhone
>
> On Sep 30, 2017, at 7:35 AM, Vega Cai  wrote:
>
> Hi,
>
> In Tricircle we use the "multinode" topology to setup a test environment
> with three regions, "CentralRegion" and "RegionOne" in one node, and
> "RegionTwo" in the other node. I notice that the job definition has been
> migrated to
> openstack-zuul-jobs/blob/master/playbooks/legacy/tricircle-dsvm-multiregion/run.yaml,
> but the job fails with the error that "public endpoint for image service in
> RegionTwo region not found", so I guess the node of "RegionTwo" is not
> correctly running. Since the original log folder for the second
> "subnode-2/" is missing in the job report, I also cannot figure out what
> the wrong is with the second node.
>
> Any hints to debug this problem?
>
>
> On Fri, 29 Sep 2017 at 22:59 Monty Taylor  wrote:
>
>> Hey everybody!
>>
>> tl;dr - If you're having issues with your jobs, check the FAQ, this
>> email and followups on this thread for mentions of them. If it's an
>> issue with your job and you can spot it (bad config) just submit a patch
>> with topic 'zuulv3'. If it's bigger/weirder/you don't know - we'd like
>> to ask that you send a follow up email to this thread so that we can
>> ensure we've got them all and so that others can see it too.
>>
>> ** Zuul v3 Migration Status **
>>
>> If you haven't noticed the Zuul v3 migration - awesome, that means it's
>> working perfectly for you.
>>
>> If you have - sorry for the disruption. It turns out we have a REALLY
>> complicated array of job content you've all created. Hopefully the pain
>> of the moment will be offset by the ability for you to all take direct
>> ownership of your awesome content... so bear with us, your patience is
>> appreciated.
>>
>> If you find yourself with some extra time on your hands while you wait
>> on something, you may find it helpful to read:
>>
>>https://docs.openstack.org/infra/manual/zuulv3.html
>>
>> We're adding content to it as issues arise. Unfortunately, one of the
>> issues is that the infra manual publication job stopped working.
>>
>> While the infra manual publication is being fixed, we're collecting FAQ
>> content for it in an etherpad:
>>
>>https://etherpad.openstack.org/p/zuulv3-migration-faq
>>
>> If you have a job issue, check it first to see if we've got an entry for
>> it. Once manual publication is fixed, we'll update the etherpad to point
>> to the FAQ section of the manual.
>>
>> ** Global Issues **
>>
>> There are a number of outstanding issues that are being worked. As of
>> right now, there are a few major/systemic ones that we're looking in to
>> that are worth noting:
>>
>> * Zuul Stalls
>>
>> If you say to yourself "zuul doesn't seem to be doing anything, did I do
>> something wrong?", we're having an issue that jeblair and Shrews are
>> currently tracking down with intermittent connection issues in the
>> backend plumbing.
>>
>> When it happens it's an across the board issue, so fixing it is our
>> number one priority.
>>
>> * Incorrect node type
>>
>> We've got reports of things running on trusty that should be running on
>> xenial. The job definitions look correct, so this is also under
>> investigation.
>>
>> * Multinode jobs having POST FAILURE
>>
>> There is a bug in the log collection trying to collect from all nodes
>> while the old jobs were designed to only collect from the 'primary'.
>> Patches are up to fix this and should be fixed soon.
>>
>> * Branch Exclusions being ignored
>>
>> This has been reported and its cause is currently unknown.
>>
>> Thank you all again for your patience! This is a giant rollout with a
>> bunch of changes in it, so we really do appreciate everyone's
>> understanding as we work through it all.
>>
>> Monty
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> --
> BR
> Zhiyuan
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>

Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-09-30 Thread Mohammed Naser 
Hi Vega,

Please check the document. Some jobs were migrated with incorrect nodesets and 
have to be switched to multinode in the job definition in openstack-zuul-jobs

Good luck
Mohammed

Sent from my iPhone

> On Sep 30, 2017, at 7:35 AM, Vega Cai  wrote:
> 
> Hi,
> 
> In Tricircle we use the "multinode" topology to setup a test environment with 
> three regions, "CentralRegion" and "RegionOne" in one node, and "RegionTwo" 
> in the other node. I notice that the job definition has been migrated to 
> openstack-zuul-jobs/blob/master/playbooks/legacy/tricircle-dsvm-multiregion/run.yaml,
>  but the job fails with the error that "public endpoint for image service in 
> RegionTwo region not found", so I guess the node of "RegionTwo" is not 
> correctly running. Since the original log folder for the second "subnode-2/" 
> is missing in the job report, I also cannot figure out what the wrong is with 
> the second node.
> 
> Any hints to debug this problem?
> 
> 
>> On Fri, 29 Sep 2017 at 22:59 Monty Taylor  wrote:
>> Hey everybody!
>> 
>> tl;dr - If you're having issues with your jobs, check the FAQ, this
>> email and followups on this thread for mentions of them. If it's an
>> issue with your job and you can spot it (bad config) just submit a patch
>> with topic 'zuulv3'. If it's bigger/weirder/you don't know - we'd like
>> to ask that you send a follow up email to this thread so that we can
>> ensure we've got them all and so that others can see it too.
>> 
>> ** Zuul v3 Migration Status **
>> 
>> If you haven't noticed the Zuul v3 migration - awesome, that means it's
>> working perfectly for you.
>> 
>> If you have - sorry for the disruption. It turns out we have a REALLY
>> complicated array of job content you've all created. Hopefully the pain
>> of the moment will be offset by the ability for you to all take direct
>> ownership of your awesome content... so bear with us, your patience is
>> appreciated.
>> 
>> If you find yourself with some extra time on your hands while you wait
>> on something, you may find it helpful to read:
>> 
>>https://docs.openstack.org/infra/manual/zuulv3.html
>> 
>> We're adding content to it as issues arise. Unfortunately, one of the
>> issues is that the infra manual publication job stopped working.
>> 
>> While the infra manual publication is being fixed, we're collecting FAQ
>> content for it in an etherpad:
>> 
>>https://etherpad.openstack.org/p/zuulv3-migration-faq
>> 
>> If you have a job issue, check it first to see if we've got an entry for
>> it. Once manual publication is fixed, we'll update the etherpad to point
>> to the FAQ section of the manual.
>> 
>> ** Global Issues **
>> 
>> There are a number of outstanding issues that are being worked. As of
>> right now, there are a few major/systemic ones that we're looking in to
>> that are worth noting:
>> 
>> * Zuul Stalls
>> 
>> If you say to yourself "zuul doesn't seem to be doing anything, did I do
>> something wrong?", we're having an issue that jeblair and Shrews are
>> currently tracking down with intermittent connection issues in the
>> backend plumbing.
>> 
>> When it happens it's an across the board issue, so fixing it is our
>> number one priority.
>> 
>> * Incorrect node type
>> 
>> We've got reports of things running on trusty that should be running on
>> xenial. The job definitions look correct, so this is also under
>> investigation.
>> 
>> * Multinode jobs having POST FAILURE
>> 
>> There is a bug in the log collection trying to collect from all nodes
>> while the old jobs were designed to only collect from the 'primary'.
>> Patches are up to fix this and should be fixed soon.
>> 
>> * Branch Exclusions being ignored
>> 
>> This has been reported and its cause is currently unknown.
>> 
>> Thank you all again for your patience! This is a giant rollout with a
>> bunch of changes in it, so we really do appreciate everyone's
>> understanding as we work through it all.
>> 
>> Monty
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> -- 
> BR
> Zhiyuan
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-09-30 Thread Vega Cai 
Hi,

In Tricircle we use the "multinode" topology to setup a test environment
with three regions, "CentralRegion" and "RegionOne" in one node, and
"RegionTwo" in the other node. I notice that the job definition has been
migrated to
openstack-zuul-jobs/blob/master/playbooks/legacy/tricircle-dsvm-multiregion/run.yaml,
but the job fails with the error that "public endpoint for image service in
RegionTwo region not found", so I guess the node of "RegionTwo" is not
correctly running. Since the original log folder for the second
"subnode-2/" is missing in the job report, I also cannot figure out what
the wrong is with the second node.

Any hints to debug this problem?


On Fri, 29 Sep 2017 at 22:59 Monty Taylor  wrote:

> Hey everybody!
>
> tl;dr - If you're having issues with your jobs, check the FAQ, this
> email and followups on this thread for mentions of them. If it's an
> issue with your job and you can spot it (bad config) just submit a patch
> with topic 'zuulv3'. If it's bigger/weirder/you don't know - we'd like
> to ask that you send a follow up email to this thread so that we can
> ensure we've got them all and so that others can see it too.
>
> ** Zuul v3 Migration Status **
>
> If you haven't noticed the Zuul v3 migration - awesome, that means it's
> working perfectly for you.
>
> If you have - sorry for the disruption. It turns out we have a REALLY
> complicated array of job content you've all created. Hopefully the pain
> of the moment will be offset by the ability for you to all take direct
> ownership of your awesome content... so bear with us, your patience is
> appreciated.
>
> If you find yourself with some extra time on your hands while you wait
> on something, you may find it helpful to read:
>
>https://docs.openstack.org/infra/manual/zuulv3.html
>
> We're adding content to it as issues arise. Unfortunately, one of the
> issues is that the infra manual publication job stopped working.
>
> While the infra manual publication is being fixed, we're collecting FAQ
> content for it in an etherpad:
>
>https://etherpad.openstack.org/p/zuulv3-migration-faq
>
> If you have a job issue, check it first to see if we've got an entry for
> it. Once manual publication is fixed, we'll update the etherpad to point
> to the FAQ section of the manual.
>
> ** Global Issues **
>
> There are a number of outstanding issues that are being worked. As of
> right now, there are a few major/systemic ones that we're looking in to
> that are worth noting:
>
> * Zuul Stalls
>
> If you say to yourself "zuul doesn't seem to be doing anything, did I do
> something wrong?", we're having an issue that jeblair and Shrews are
> currently tracking down with intermittent connection issues in the
> backend plumbing.
>
> When it happens it's an across the board issue, so fixing it is our
> number one priority.
>
> * Incorrect node type
>
> We've got reports of things running on trusty that should be running on
> xenial. The job definitions look correct, so this is also under
> investigation.
>
> * Multinode jobs having POST FAILURE
>
> There is a bug in the log collection trying to collect from all nodes
> while the old jobs were designed to only collect from the 'primary'.
> Patches are up to fix this and should be fixed soon.
>
> * Branch Exclusions being ignored
>
> This has been reported and its cause is currently unknown.
>
> Thank you all again for your patience! This is a giant rollout with a
> bunch of changes in it, so we really do appreciate everyone's
> understanding as we work through it all.
>
> Monty
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 
BR
Zhiyuan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Update on Zuul v3 Migration - and what to do about issues

2017-09-29 Thread Monty Taylor 

Hey everybody!

tl;dr - If you're having issues with your jobs, check the FAQ, this 
email and followups on this thread for mentions of them. If it's an 
issue with your job and you can spot it (bad config) just submit a patch 
with topic 'zuulv3'. If it's bigger/weirder/you don't know - we'd like 
to ask that you send a follow up email to this thread so that we can 
ensure we've got them all and so that others can see it too.


** Zuul v3 Migration Status **

If you haven't noticed the Zuul v3 migration - awesome, that means it's 
working perfectly for you.


If you have - sorry for the disruption. It turns out we have a REALLY 
complicated array of job content you've all created. Hopefully the pain 
of the moment will be offset by the ability for you to all take direct 
ownership of your awesome content... so bear with us, your patience is 
appreciated.


If you find yourself with some extra time on your hands while you wait 
on something, you may find it helpful to read:


  https://docs.openstack.org/infra/manual/zuulv3.html

We're adding content to it as issues arise. Unfortunately, one of the 
issues is that the infra manual publication job stopped working.


While the infra manual publication is being fixed, we're collecting FAQ 
content for it in an etherpad:


  https://etherpad.openstack.org/p/zuulv3-migration-faq

If you have a job issue, check it first to see if we've got an entry for 
it. Once manual publication is fixed, we'll update the etherpad to point 
to the FAQ section of the manual.


** Global Issues **

There are a number of outstanding issues that are being worked. As of 
right now, there are a few major/systemic ones that we're looking in to 
that are worth noting:


* Zuul Stalls

If you say to yourself "zuul doesn't seem to be doing anything, did I do 
something wrong?", we're having an issue that jeblair and Shrews are 
currently tracking down with intermittent connection issues in the 
backend plumbing.


When it happens it's an across the board issue, so fixing it is our 
number one priority.


* Incorrect node type

We've got reports of things running on trusty that should be running on 
xenial. The job definitions look correct, so this is also under 
investigation.


* Multinode jobs having POST FAILURE

There is a bug in the log collection trying to collect from all nodes 
while the old jobs were designed to only collect from the 'primary'. 
Patches are up to fix this and should be fixed soon.


* Branch Exclusions being ignored

This has been reported and its cause is currently unknown.

Thank you all again for your patience! This is a giant rollout with a 
bunch of changes in it, so we really do appreciate everyone's 
understanding as we work through it all.


Monty

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev