Any update on where we stand on issues now? Because every single patch I tried to land yesterday was killed by POST_FAILURE in various ways. Including some really small stuff - https://review.openstack.org/#/c/324720/
That also includes the patch I'm told fixes some issues with zuul v3 in the base devstack jobs - https://review.openstack.org/#/c/508344/3 It also appears that many of the skips stopped being a thing - https://review.openstack.org/#/c/507527/ got a Tempest test run attempted on it (though everything ended in Node failure). Do we have a defined point on the calendar for getting the false negatives back below the noise threshold otherwise a rollback is implemented so that some of these issues can be addressed in parallel without holding up community development? -Sean On 09/29/2017 10:58 AM, Monty Taylor wrote: > Hey everybody! > > tl;dr - If you're having issues with your jobs, check the FAQ, this > email and followups on this thread for mentions of them. If it's an > issue with your job and you can spot it (bad config) just submit a patch > with topic 'zuulv3'. If it's bigger/weirder/you don't know - we'd like > to ask that you send a follow up email to this thread so that we can > ensure we've got them all and so that others can see it too. > > ** Zuul v3 Migration Status ** > > If you haven't noticed the Zuul v3 migration - awesome, that means it's > working perfectly for you. > > If you have - sorry for the disruption. It turns out we have a REALLY > complicated array of job content you've all created. Hopefully the pain > of the moment will be offset by the ability for you to all take direct > ownership of your awesome content... so bear with us, your patience is > appreciated. > > If you find yourself with some extra time on your hands while you wait > on something, you may find it helpful to read: > > https://docs.openstack.org/infra/manual/zuulv3.html > > We're adding content to it as issues arise. Unfortunately, one of the > issues is that the infra manual publication job stopped working. > > While the infra manual publication is being fixed, we're collecting FAQ > content for it in an etherpad: > > https://etherpad.openstack.org/p/zuulv3-migration-faq > > If you have a job issue, check it first to see if we've got an entry for > it. Once manual publication is fixed, we'll update the etherpad to point > to the FAQ section of the manual. > > ** Global Issues ** > > There are a number of outstanding issues that are being worked. As of > right now, there are a few major/systemic ones that we're looking in to > that are worth noting: > > * Zuul Stalls > > If you say to yourself "zuul doesn't seem to be doing anything, did I do > something wrong?", we're having an issue that jeblair and Shrews are > currently tracking down with intermittent connection issues in the > backend plumbing. > > When it happens it's an across the board issue, so fixing it is our > number one priority. > > * Incorrect node type > > We've got reports of things running on trusty that should be running on > xenial. The job definitions look correct, so this is also under > investigation. > > * Multinode jobs having POST FAILURE > > There is a bug in the log collection trying to collect from all nodes > while the old jobs were designed to only collect from the 'primary'. > Patches are up to fix this and should be fixed soon. > > * Branch Exclusions being ignored > > This has been reported and its cause is currently unknown. > > Thank you all again for your patience! This is a giant rollout with a > bunch of changes in it, so we really do appreciate everyone's > understanding as we work through it all. > > Monty > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sean Dague http://dague.net __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev