Re: [openstack-dev] [nova] Online Migrations.
There are multiple nova objects that each have their own version and each object corresponds to a database model. I think this might be the best solution for being able to determine if rows are migrated or not. My basic thought is that each table has a the corresponding object class noted in the .info dict for the table allowing us to determine what object class should be used to verify migrations. The biggest concern would be to verify that all rows are migrated to a point that they can all be tagged with the latest object version at the time of adding the version tag to each row. Then if this method is to be used we can use this in the future to determine if migrations of a row have completed. -Ph > On Jun 16, 2015, at 9:16 AM, Mike Bayer wrote: > > > > On 6/15/15 8:34 PM, Philip Schwartz wrote: >> I discussed this a bit earlier with John and we came up with a thought that >> I was going to present after getting a little bit more documentation and >> spec around. With out going into too much detail, here is the basics of the >> idea. >> >> Add a new column to all data models that allow us to inject with >> insert/update of rows the version of the Nova object it is for. Then we can >> add logic that prevents the contract from being run till a condition is met >> for a specific period of time after an object version has been deprecated. >> Once the depreciation window passes, it would be safe to remove the column >> form the model and contract the DB. This fits with our current thinking and >> the ability for conductor to down cast objects to older object versions and >> best of all, it is easy for us to maintain and access as the version for >> each row creation has access to the nova object and the version set in the >> object class. >> >> If we set the criteria for breaking backwards compatibility and object >> downgrading with a new major version `VERSION = ‘2.0’` we know at that point >> it is safe to remove columns from the model that became deprecated prior to >> ‘2.0’ and allow the contract to run as long as all rows of data have a >> version in them of ‘2.0’. >> >> This does not have to be a major version and could really just be an >> arbitrary object version + N that we decide as a community. > > How much of a 1-1 relationship is there from database table -> Nova object ? > To what extent does this change enforce that 1-1 vs. remaining agnostic of > it? I ask because one of the issues some of us see with the objects approach > is that it can be taxing on performance and flexibility if it exposes an API > that is too fine-grained and molded to the structure of tables. > > > > >> >> -Ph >> >>> On Jun 15, 2015, at 8:06 PM, Mike Bayer wrote: >>> >>> >>> >>> On 6/15/15 6:37 PM, Mike Bayer wrote: On 6/15/15 4:21 PM, Andrew Laski wrote: If I had to visualize what an approach looks like that does this somewhat cleanly, other than just putting off contract until the API has naturally moved beyond it, it would involve a fixed and structured source of truth about the specific changes we care about, such as a versioning table or other data table indicating specific "remove()" directives we're checking for, and the application would be organized such that it can always get to this information from an in-memory-cached source before it makes decisions about queries. The information would need to support being pushed in from the outside such as via a message queue. This would still not protect against operations currently in progress failing but at least would prevent future operations from failing a first time. >>> Or, what I was thinking earlier before I focused too deeply on this whole >>> thing, you basically get all running applications to no longer talk to the >>> to-be-removed structures at all first, *then* do the contract. >>> >>> That is, you're on version L. You've done your expand, you're running the >>> multi-schema version of the model. All your data is migrated.Now some >>> config flag or something else changes somewhere (still need to work out >>> this part), which says, "we're done with all the removed() columns". All >>> the apps ultimately get restarted with this new flag in place - the whole >>> thing is now running without including removed() columns in the model >>> (they're still there in the source code, but as I illustrated earlier, some >>> conditional logic has prevented them from actually being part of the model >>> on this new run). >>> >>> *Then* you run the contract. Then you don't have to worry about runtime >>> failures or tracking specific columns or any of that. There's just some >>> kind of state that indicates, "ready for L contract". It's still >>> something of a "version" but it is local to a single version of the >>> software; instead of waiting for a full upgrade from version L to M, you >>> have th
Re: [openstack-dev] [nova] Online Migrations.
On 6/15/15 8:34 PM, Philip Schwartz wrote: I discussed this a bit earlier with John and we came up with a thought that I was going to present after getting a little bit more documentation and spec around. With out going into too much detail, here is the basics of the idea. Add a new column to all data models that allow us to inject with insert/update of rows the version of the Nova object it is for. Then we can add logic that prevents the contract from being run till a condition is met for a specific period of time after an object version has been deprecated. Once the depreciation window passes, it would be safe to remove the column form the model and contract the DB. This fits with our current thinking and the ability for conductor to down cast objects to older object versions and best of all, it is easy for us to maintain and access as the version for each row creation has access to the nova object and the version set in the object class. If we set the criteria for breaking backwards compatibility and object downgrading with a new major version `VERSION = ‘2.0’` we know at that point it is safe to remove columns from the model that became deprecated prior to ‘2.0’ and allow the contract to run as long as all rows of data have a version in them of ‘2.0’. This does not have to be a major version and could really just be an arbitrary object version + N that we decide as a community. How much of a 1-1 relationship is there from database table -> Nova object ?To what extent does this change enforce that 1-1 vs. remaining agnostic of it? I ask because one of the issues some of us see with the objects approach is that it can be taxing on performance and flexibility if it exposes an API that is too fine-grained and molded to the structure of tables. -Ph On Jun 15, 2015, at 8:06 PM, Mike Bayer wrote: On 6/15/15 6:37 PM, Mike Bayer wrote: On 6/15/15 4:21 PM, Andrew Laski wrote: If I had to visualize what an approach looks like that does this somewhat cleanly, other than just putting off contract until the API has naturally moved beyond it, it would involve a fixed and structured source of truth about the specific changes we care about, such as a versioning table or other data table indicating specific "remove()" directives we're checking for, and the application would be organized such that it can always get to this information from an in-memory-cached source before it makes decisions about queries. The information would need to support being pushed in from the outside such as via a message queue. This would still not protect against operations currently in progress failing but at least would prevent future operations from failing a first time. Or, what I was thinking earlier before I focused too deeply on this whole thing, you basically get all running applications to no longer talk to the to-be-removed structures at all first, *then* do the contract. That is, you're on version L. You've done your expand, you're running the multi-schema version of the model. All your data is migrated.Now some config flag or something else changes somewhere (still need to work out this part), which says, "we're done with all the removed() columns". All the apps ultimately get restarted with this new flag in place - the whole thing is now running without including removed() columns in the model (they're still there in the source code, but as I illustrated earlier, some conditional logic has prevented them from actually being part of the model on this new run). *Then* you run the contract. Then you don't have to worry about runtime failures or tracking specific columns or any of that. There's just some kind of state that indicates, "ready for L contract". It's still something of a "version" but it is local to a single version of the software; instead of waiting for a full upgrade from version L to M, you have this internal state that can somehow move from L(m) to L(c).That is a lot more doable and sane than trying to guess at startup / runtime what columns are being yanked. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Online Migrations.
I discussed this a bit earlier with John and we came up with a thought that I was going to present after getting a little bit more documentation and spec around. With out going into too much detail, here is the basics of the idea. Add a new column to all data models that allow us to inject with insert/update of rows the version of the Nova object it is for. Then we can add logic that prevents the contract from being run till a condition is met for a specific period of time after an object version has been deprecated. Once the depreciation window passes, it would be safe to remove the column form the model and contract the DB. This fits with our current thinking and the ability for conductor to down cast objects to older object versions and best of all, it is easy for us to maintain and access as the version for each row creation has access to the nova object and the version set in the object class. If we set the criteria for breaking backwards compatibility and object downgrading with a new major version `VERSION = ‘2.0’` we know at that point it is safe to remove columns from the model that became deprecated prior to ‘2.0’ and allow the contract to run as long as all rows of data have a version in them of ‘2.0’. This does not have to be a major version and could really just be an arbitrary object version + N that we decide as a community. -Ph > On Jun 15, 2015, at 8:06 PM, Mike Bayer wrote: > > > > On 6/15/15 6:37 PM, Mike Bayer wrote: >> >> >> On 6/15/15 4:21 PM, Andrew Laski wrote: >> >> If I had to visualize what an approach looks like that does this somewhat >> cleanly, other than just putting off contract until the API has naturally >> moved beyond it, it would involve a fixed and structured source of truth >> about the specific changes we care about, such as a versioning table or >> other data table indicating specific "remove()" directives we're checking >> for, and the application would be organized such that it can always get to >> this information from an in-memory-cached source before it makes decisions >> about queries. The information would need to support being pushed in from >> the outside such as via a message queue. This would still not protect >> against operations currently in progress failing but at least would prevent >> future operations from failing a first time. >> > > Or, what I was thinking earlier before I focused too deeply on this whole > thing, you basically get all running applications to no longer talk to the > to-be-removed structures at all first, *then* do the contract. > > That is, you're on version L. You've done your expand, you're running the > multi-schema version of the model. All your data is migrated.Now some > config flag or something else changes somewhere (still need to work out this > part), which says, "we're done with all the removed() columns". All the > apps ultimately get restarted with this new flag in place - the whole thing > is now running without including removed() columns in the model (they're > still there in the source code, but as I illustrated earlier, some > conditional logic has prevented them from actually being part of the model on > this new run). > > *Then* you run the contract. Then you don't have to worry about runtime > failures or tracking specific columns or any of that. There's just some kind > of state that indicates, "ready for L contract". It's still something of a > "version" but it is local to a single version of the software; instead of > waiting for a full upgrade from version L to M, you have this internal state > that can somehow move from L(m) to L(c).That is a lot more doable and > sane than trying to guess at startup / runtime what columns are being yanked. > > > > > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
