On Tue, May 9, 2017 at 11:46 AM, mathieu bultel <mbul...@redhat.com> wrote:
> On 05/08/2017 01:45 PM, Marios Andreou wrote: > > Hi folks, after some discussion locally with colleagues about improving > the upgrades experience, one of the items that came up was pre-upgrade and > update validations. I took an AI to look at the current status of > tripleo-validations [0] and posted a simple WIP [1] intended to be run > before an undercloud update/upgrade and which just checks service status. > It was pointed out by shardy that for such checks it is better to instead > continue to use the per-service manifests where possible like [2] for > example where we check status before N..O major upgrade. There may still be > some undercloud specific validations that we can land into the > tripleo-validations repo (thinking about things like the neutron > networks/ports, validating the current nova nodes state etc?). > > Yes, I think a bunch of validation: db states, services states, network > connectivity (external, internal) > > > So do folks have any thoughts about this subject - for example the kinds > of things we should be checking - Steve said he had some reviews in > progress for collecting the overcloud ansible puppet/docker config into an > ansible playbook that the operator can invoke for upgrade of the 'manual' > nodes (for example compute in the N..O workflow) - the point being that we > can add more per-service ansible validation tasks into the service > manifests for execution when the play is run by the operator - but I'll let > Steve point at and talk about those. > > I have a WIP review about that [1], but i need to revisit it a bit, to add > a part into the mistral workflow (Im also writing a POC to create a mistral > workbook for major upgrade and validate minor/major upgrade before starting > [2], I have a third one in progress, not pushed yet, to implement the major > upgrade option in the cli): > > [1] https://review.openstack.org/444224 <https://review.openstack.org/444224> > [2] https://review.openstack.org/#/c/462961 > > > ack I had a pass at those when you first sent this - will look again. I think our discussion have highlighted a few things * lack of tripleo-common/client support for upgrades workflow (e.g. so we can kill the -e for every invocation) * better - more pre-upgrade/update validations - especially undercloud doesn't have much/any (already have some pre-upgrade validations) also minor update doesn't have much/any converage . * improving the 'manual node upgrade' - instead of running upgrade-non-controller.sh use a playbook. that can be run by the operator (or even automated?) * better logging tracking of current upgrades step/progress - when failures happen * related to ^^^ when upgrade/update completes on current node writing a /etc/tripleorelease or somesuch to say 'mitaka' or 'ocata' or whatever the version just been upgraded to, is. I recall Emilien suggesting blueprint for tracking we should discuss some more and get these down as one or more blueprints probably - assuming folks agree with that list and what we can add/remove/change - lets work together to split the blueprints if you like but lets discuss them a bit here first/or not more comments lets see :) thanks > cheers, marios > > [0] https://github.com/openstack/tripleo-validations > [1] https://review.openstack.org/#/c/462918/ > [2] <https://github.com/openstack/>https://github.com/openstack/ > tripleo-heat-templates/blob/stable/ocata/puppet/services/neutron-api.yaml# > L197 > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev