Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

in the DB via a two-way trigger. Regards, Alex -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: Friday, October 14, 2016 1:56 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 14 October 2016 at 23:55, Jay Pipes wrote: > The primary thing that, to me at least, differentiates rolling upgrades of > distributed software is that different nodes can contain multiple versions > of the software and continue to communicate with other nodes in the system

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Excerpts from Jay Pipes's message of 2016-10-14 16:55:39 -0400: > Alex, so sorry for the long delayed response! :( This just crept to > the back of my inbox unfortunately. Answer inline... > > On 09/14/2016 07:24 PM, Bashmakov, Alexander wrote: > >> Glance and Keystone do not participate in a

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Alex, so sorry for the long delayed response! :( This just crept to the back of my inbox unfortunately. Answer inline... On 09/14/2016 07:24 PM, Bashmakov, Alexander wrote: Glance and Keystone do not participate in a rolling upgrade, because Keystone and Glance do not have a distributed

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Excerpts from Henry Nash's message of 2016-09-15 00:29:44 +0100: > Jay, > > I agree with your distinction - and when I am referring to rolling upgrades > for keystone I am referring to when you are running a cluster of keystones > (for performance and/or redundancy), and you want to roll the

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Jay, I agree with your distinction - and when I am referring to rolling upgrades for keystone I am referring to when you are running a cluster of keystones (for performance and/or redundancy), and you want to roll the upgrade across the cluster without creating downtime of the overall keystone

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

> Glance and Keystone do not participate in a rolling upgrade, because > Keystone and Glance do not have a distributed component architecture. > Online data migrations will reduce total downtime experienced during an > *overall upgrade procedure* for an OpenStack cloud, but Nova, Neutron and >

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 09/01/2016 05:29 AM, Henry Nash wrote: So as the person who drove the rolling upgrade requirements into keystone in this cycle (because we have real customers that need it), and having first written the keystone upgrade process to be “versioned object ready” (because I assumed we would do

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Excerpts from Mike Bayer's message of 2016-09-02 17:58:42 -0400: > > On 09/02/2016 01:53 PM, Doug Hellmann wrote: > > Excerpts from Thierry Carrez's message of 2016-09-02 12:15:33 +0200: > >> Sean Dague wrote: > >>> Putting DB trigger failure analysis into the toolkit required to manage > >>> an

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

> > On 09/02/2016 01:53 PM, Doug Hellmann wrote: > >> Excerpts from Thierry Carrez's message of 2016-09-02 12:15:33 +0200: > > I agree with Sean: increasing the variety of technologies used increases >>> the system complexity, which in turn requires more skills to fully >>> understand and maintain

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 09/02/2016 01:53 PM, Doug Hellmann wrote: Excerpts from Thierry Carrez's message of 2016-09-02 12:15:33 +0200: Sean Dague wrote: Putting DB trigger failure analysis into the toolkit required to manage an upgrade failure is a really high bar for new ops. I agree with Sean: increasing the

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Excerpts from Thierry Carrez's message of 2016-09-02 12:15:33 +0200: > Sean Dague wrote: > > Putting DB trigger failure analysis into the toolkit required to manage > > an upgrade failure is a really high bar for new ops. > > I agree with Sean: increasing the variety of technologies used

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Sean Dague wrote: > Putting DB trigger failure analysis into the toolkit required to manage > an upgrade failure is a really high bar for new ops. I agree with Sean: increasing the variety of technologies used increases the system complexity, which in turn requires more skills to fully understand

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On Thursday, September 1, 2016, Jeremy Stanley wrote: > > I don't read that at all as suggesting "the problem is solved, go > away" but rather "help us make it better for everyone, don't just > take one project off in a new direction and leave the others > behind." I can

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Excerpts from Robert Collins's message of 2016-09-01 20:45:22 +1200: > On 31 August 2016 at 01:57, Clint Byrum wrote: > > > > > > It's simple, these are the holy SQL schema commandments: > > > > Don't delete columns, ignore them. > > Don't change columns, create new ones. > >

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 2016-09-01 10:39:09 -0400 (-0400), Mike Bayer wrote: > On 08/31/2016 06:18 PM, Monty Taylor wrote: [...] > >OpenStack is One Project > > > > > > Nova and Neutron have an approach for this. It may or may not be > > ideal - but it exists right now. While it can be

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 09/01/2016 11:52 AM, Dan Smith wrote: The indirection service is really unrelated to this discussion, IMHO. If you take RPC out of the picture, all you have left is a direct-to-the-database facade to handle the fact that schema has expanded underneath you. As Clint (et al) have said --

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

> So that is fine. However, correct me if I'm wrong but you're > proposing just that these projects migrate to also use a new service > layer with oslo.versionedobjects, because IIUC Nova/Neutron's > approach is dependent on that area of indirection being present. > Otherwise, if you meant

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 09/01/2016 09:45 AM, David Stanek wrote: > On Thu, Aug 25 at 13:13 -0400, Steve Martinelli wrote: >> The keystone team is pursuing a trigger-based approach to support rolling, >> zero-downtime upgrades. The proposed operator experience is documented here: >> >>

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 09/01/2016 08:29 AM, Henry Nash wrote: From a purely keystone perspective, my gut feeling is that actually the trigger approach is likely to lead to a more robust, not less, solution - due to the fact that we solve the very specific problems of a given migration (i.e. need to keep column A

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 08/31/2016 06:18 PM, Monty Taylor wrote: I said this the other day in the IRC channel, and I'm going to say it again here. I'm going to do it as bluntly as I can - please keeping in mind that I respect all of the humans involved. I think this is a monstrously terrible idea. There are

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On Wed, Aug 31 at 17:18 -0500, Monty Taylor wrote: > > Nova and Neutron have an approach for this. It may or may not be ideal - > but it exists right now. While it can be satisfying to discount the > existing approach and write a new one, I do not believe that is in the > best interests of

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On Thu, Aug 25 at 13:13 -0400, Steve Martinelli wrote: > The keystone team is pursuing a trigger-based approach to support rolling, > zero-downtime upgrades. The proposed operator experience is documented here: > > http://docs.openstack.org/developer/keystone/upgrading.html > I wanted to

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

So as the person who drove the rolling upgrade requirements into keystone in this cycle (because we have real customers that need it), and having first written the keystone upgrade process to be “versioned object ready” (because I assumed we would do this the same as everyone else), and

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 31 August 2016 at 01:57, Clint Byrum wrote: > > > It's simple, these are the holy SQL schema commandments: > > Don't delete columns, ignore them. > Don't change columns, create new ones. > When you create a column, give it a default that makes sense. I'm sure you're aware of

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 08/25/2016 04:14 PM, Sean Dague wrote: > On 08/25/2016 01:13 PM, Steve Martinelli wrote: >> The keystone team is pursuing a trigger-based approach to support >> rolling, zero-downtime upgrades. The proposed operator experience is >> documented here: >> >>

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 08/30/2016 08:04 PM, Clint Byrum wrote: My direct experience with this was MySQL 5.0 and 5.1. They worked as documented, and no I don't think they've changed much since then. When they were actually installed into the schema and up to date with the code that expected them, and the

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Excerpts from Mike Bayer's message of 2016-08-30 18:15:14 -0400: > > On 08/30/2016 04:43 PM, Clint Byrum wrote: > >> > > > > Correct, it is harder for development. Since the database server has all > > of the potential for the worst problems, being a stateful service, then > > I believe moving

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Since the encrypted credential work is currently based on triggers, I spent most of today documenting a walk-though migration from Mitaka to Newton [0]. Regardless of the outcome discussed here - figured it would be worth sharing since it's relevant to the thread. Most of the gist contains stuff

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 08/30/2016 04:43 PM, Clint Byrum wrote: Correct, it is harder for development. Since the database server has all of the potential for the worst problems, being a stateful service, then I believe moving complexity _out_ of it, is generally an operational win, at the expense of some

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Excerpts from Mike Bayer's message of 2016-08-30 14:56:15 -0400: > > On 08/30/2016 09:57 AM, Clint Byrum wrote: > >> > > > > As someone else brought up, this is an unnecessarily bleak view of how > > database > > migrations work. > > We aren't talking about database migrations. We are talking

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

>> I don't think it's all that ambitious to think we can just use >> tried and tested schema evolution techniques that work for everyone >> else. > > People have been asking me for over a year how to do this, and I have > no easy answer, I'm glad that you do. I would like to see some > examples

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 08/30/2016 09:57 AM, Clint Byrum wrote: As someone else brought up, this is an unnecessarily bleak view of how database migrations work. We aren't talking about database migrations. We are talking about *online* database migrations, where we would like both the *old* and *new*

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

>> Even in the case of projects using versioned objects, it still >> means a SQL layer has to include functionality for both versions of >> a particular schema change which itself is awkward. That's not true. Nova doesn't have multiple models to straddle a particular change. We just... > It's

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Excerpts from Mike Bayer's message of 2016-08-26 11:50:24 -0400: > > On 08/25/2016 01:13 PM, Steve Martinelli wrote: > > The keystone team is pursuing a trigger-based approach to support > > rolling, zero-downtime upgrades. The proposed operator experience is > > documented here: > > > >

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 25/08/16 13:13 -0400, Steve Martinelli wrote: The keystone team is pursuing a trigger-based approach to support rolling, zero-downtime upgrades. The proposed operator experience is documented here: http://docs.openstack.org/developer/keystone/upgrading.html This differs from Nova and

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 08/25/2016 01:13 PM, Steve Martinelli wrote: The keystone team is pursuing a trigger-based approach to support rolling, zero-downtime upgrades. The proposed operator experience is documented here: http://docs.openstack.org/developer/keystone/upgrading.html This differs from Nova and

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 08/25/2016 01:13 PM, Steve Martinelli wrote: The keystone team is pursuing a trigger-based approach to support rolling, zero-downtime upgrades. The proposed operator experience is documented here: http://docs.openstack.org/developer/keystone/upgrading.html This differs from Nova and

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On 25/08/16 01:13 PM, Steve Martinelli wrote: The keystone team is pursuing a trigger-based approach to support rolling, zero-downtime upgrades. The proposed operator experience is documented here: http://docs.openstack.org/developer/keystone/upgrading.html This differs from Nova and

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

> This differs from Nova and Neutron's approaches to solve for rolling > upgrades (which use oslo.versionedobjects), however Keystone is one of > the few services that doesn't need to manage communication between > multiple releases of multiple service components talking over the > message bus

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

On Thu, Aug 25, 2016 at 1:13 PM, Steve Martinelli wrote: > The keystone team is pursuing a trigger-based approach to support rolling, > zero-downtime upgrades. The proposed operator experience is documented here: > >

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Personally I had very bad experiences with stored procedures and triggers in previous jobs, where the amount of side effects that occurred and the overall lack of maintainability of triggers and stored procedures scared me off. We handed off changes to stored procedures and triggers to the DBAs,

[openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

The keystone team is pursuing a trigger-based approach to support rolling, zero-downtime upgrades. The proposed operator experience is documented here: http://docs.openstack.org/developer/keystone/upgrading.html This differs from Nova and Neutron's approaches to solve for rolling upgrades