Re: [openstack-dev] [Neutron] Migrations, service plugins and Grenade jobs
On Apr 14, 2014, at 11:54 AM, Salvatore Orlando mailto:sorla...@nicira.com>> wrote: 1) Specify that all migrations must run for every plugin (*) unless they are really introducing schemas which are specific to a particular technology (such as uuid mappings between neutron and backed) This approach will probably ensure all the important migrations are executed. However, it is not back portable, unless deployers decide to tear down and rebuild their databases from scratch, which is unlikely to happen. To this aim "recovery" scripts might be provided to sync up the schema state of specific service plugins with the current alembic migration registered in the neutron database; or appropriate migrations can be added in the path to fix database schemas. (*) Neutron does not address, and probably never will address, switching from plugin "a" to "b" for a given service (e.g.: core features). This option has several additional problems because of the way plugins/drivers are conditionally imported and packaged. As a result, auto generation of schema (inadvertent drop for schema) or even validation of existing models vs the the migration will fail because models are not imported unless part of the configuration. mark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Neutron Migrations, service plugins and Grenade jobs
I know alembic is designed to be global, but could we extend it to track multiple histories for a given database. In other words, various branches for different namespaces on a single database. Would this feature ameliorate the issues? Amir On Apr 15, 2014, at 8:24 AM, Kyle Mestery mailto:mest...@noironetworks.com>> wrote: On Tue, Apr 15, 2014 at 7:03 AM, Salvatore Orlando mailto:sorla...@nicira.com>> wrote: Thanks Anna. I've been following the issue so far, but I am happy to hand it over to you. I think the problem assessment is complete, but if you have more questions ping me on IRC. Regarding the solution, I think we already have a fairly wide consensus on the approach. There are however a few details to discuss: - Conflicting schemas. For instance two migrations for two distinct plugins might create tables with the same name but different columns. We first need to look at existing migrations to verify where this condition occurs, and then study a solution case by case. - Logic for "corrective" migrations. For instance a corrective migration for 569e98a8132b_metering is needed. However, such corrective migration should have logic for understanding whether the original migration has been executed or not. - Corrective actions for corrupted schemas. This would be the case, for instance, of somebody which enables metering while the database is at a migration rev higher than the one when metering was introduced. I reckon it might be the case of putting together a specification and push it to the newly created neutron-specs repo, assuming that we feel confident enough to start using this new process (Kyle and Mark might chime in on this point). Also, I would like to see this work completed by Juno-1, which I reckon is a reasonable target. I'm working to get this new specification approval process ready, hopefully by later today. Once this is done, I agree with Salvatore, pushing a gerrit review with the specification for this work will be the right approach. Of course I'm available for discussing design, implementation, reviewing and writing code. Thanks to Anna and Salvatore for taking this up! Kyle Salvatore On 15 April 2014 12:44, Anna Kamyshnikova mailto:akamyshnik...@mirantis.com>> wrote: Hello everyone! I would like to try to solve this problem. I registered blueprint on this topic https://blueprints.launchpad.net/neutron/+spec/neutron-robust-db-migrations and I'm going to experiment with options to solve this. I'm welcome any suggestions and ready to talk about it via IRC (akamyshnikova). Regards Ann On Mon, Apr 14, 2014 at 9:39 PM, Sean Dague wrote: On 04/14/2014 12:46 PM, Eugene Nikanorov wrote: Hi Salvatore, The described problem could be even worse if vendor drivers are considered. Doesn't #1 require that all DB tables are named differently? Otherwise it seems that user can't be sure in DB schema even if all tables are present. I think the big part of the problem is that we need to support both online and offline migrations. Without the latter things could be a little bit simpler. Also it seems to me that problem statement should be changed to the following: One need to migrate from (Config1, MigrationID1) to (Config2, MigrationID2), and currently our code only accounts for MigrationIDs. We may consider amending DB with configuration metadata, at least that will allow to run migration code with full knowledge of what happened before (if online mode is considered). In offline mode that will require providing old and current configurations. That was just thinking aloud, no concrete proposals yet. The root issue really is Migrations *must* be global, and config invariant. That's the design point in both sqlalchemy-migrate and alembic. The fact that there is one global migrations table per database, with a single value in it, is indicative of this fact. I think that design point got lost somewhere along the way, and folks assumed migrations were just a way to change schemas. They are much more constrained than that. It does also sound like the data model is going to need some serious reconsidering given what's allowed to be changed at the plugin or vendor driver model. Contrast this with Nova, were virt drivers don't get to define persistant data that's unique to them (only generic data that they fit into the grander nova model). The one time we had a driver which needed persistent data (baremetal) it managed it's own database entirely. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing l
Re: [openstack-dev] Neutron Migrations, service plugins and Grenade jobs
On Tue, Apr 15, 2014 at 7:03 AM, Salvatore Orlando wrote: > Thanks Anna. > > I've been following the issue so far, but I am happy to hand it over to you. > I think the problem assessment is complete, but if you have more questions > ping me on IRC. > > Regarding the solution, I think we already have a fairly wide consensus on > the approach. > There are however a few details to discuss: > - Conflicting schemas. For instance two migrations for two distinct plugins > might create tables with the same name but different columns. > We first need to look at existing migrations to verify where this > condition occurs, and then study a solution case by case. > - Logic for "corrective" migrations. For instance a corrective migration for > 569e98a8132b_metering is needed. However, such corrective migration should > have logic for understanding whether the original migration has been > executed or not. > - Corrective actions for corrupted schemas. This would be the case, for > instance, of somebody which enables metering while the database is at a > migration rev higher than the one when metering was introduced. > > I reckon it might be the case of putting together a specification and push > it to the newly created neutron-specs repo, assuming that we feel confident > enough to start using this new process (Kyle and Mark might chime in on this > point). Also, I would like to see this work completed by Juno-1, which I > reckon is a reasonable target. > I'm working to get this new specification approval process ready, hopefully by later today. Once this is done, I agree with Salvatore, pushing a gerrit review with the specification for this work will be the right approach. > Of course I'm available for discussing design, implementation, reviewing and > writing code. > Thanks to Anna and Salvatore for taking this up! Kyle > Salvatore > > > > On 15 April 2014 12:44, Anna Kamyshnikova > wrote: >> >> Hello everyone! >> >> I would like to try to solve this problem. I registered blueprint on this >> topic >> https://blueprints.launchpad.net/neutron/+spec/neutron-robust-db-migrations >> and I'm going to experiment with options to solve this. I'm welcome any >> suggestions and ready to talk about it via IRC (akamyshnikova). >> >> Regards >> Ann >> >> >> On Mon, Apr 14, 2014 at 9:39 PM, Sean Dague wrote: >>> >>> On 04/14/2014 12:46 PM, Eugene Nikanorov wrote: >>> > Hi Salvatore, >>> > >>> > The described problem could be even worse if vendor drivers are >>> > considered. >>> > Doesn't #1 require that all DB tables are named differently? Otherwise >>> > it seems that user can't be sure in DB schema even if all tables are >>> > present. >>> > >>> > I think the big part of the problem is that we need to support both >>> > online and offline migrations. Without the latter things could be a >>> > little bit simpler. >>> > >>> > Also it seems to me that problem statement should be changed to the >>> > following: >>> > One need to migrate from (Config1, MigrationID1) to (Config2, >>> > MigrationID2), and currently our code only accounts for MigrationIDs. >>> > We may consider amending DB with configuration metadata, at least that >>> > will allow to run migration code with full knowledge of what happened >>> > before (if online mode is considered). >>> > In offline mode that will require providing old and current >>> > configurations. >>> > >>> > That was just thinking aloud, no concrete proposals yet. >>> >>> The root issue really is Migrations *must* be global, and config >>> invariant. That's the design point in both sqlalchemy-migrate and >>> alembic. The fact that there is one global migrations table per >>> database, with a single value in it, is indicative of this fact. >>> >>> I think that design point got lost somewhere along the way, and folks >>> assumed migrations were just a way to change schemas. They are much more >>> constrained than that. >>> >>> It does also sound like the data model is going to need some serious >>> reconsidering given what's allowed to be changed at the plugin or vendor >>> driver model. Contrast this with Nova, were virt drivers don't get to >>> define persistant data that's unique to them (only generic data that >>> they fit into the grander nova model). >>> >>> The one time we had a driver which needed persistent data (baremetal) it >>> managed it's own database entirely. >>> >>> -Sean >>> >>> -- >>> Sean Dague >>> Samsung Research America >>> s...@dague.net / sean.da...@samsung.com >>> http://dague.net >>> >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list >
Re: [openstack-dev] Neutron Migrations, service plugins and Grenade jobs
Thanks Anna. I've been following the issue so far, but I am happy to hand it over to you. I think the problem assessment is complete, but if you have more questions ping me on IRC. Regarding the solution, I think we already have a fairly wide consensus on the approach. There are however a few details to discuss: - Conflicting schemas. For instance two migrations for two distinct plugins might create tables with the same name but different columns. We first need to look at existing migrations to verify where this condition occurs, and then study a solution case by case. - Logic for "corrective" migrations. For instance a corrective migration for 569e98a8132b_metering is needed. However, such corrective migration should have logic for understanding whether the original migration has been executed or not. - Corrective actions for corrupted schemas. This would be the case, for instance, of somebody which enables metering while the database is at a migration rev higher than the one when metering was introduced. I reckon it might be the case of putting together a specification and push it to the newly created neutron-specs repo, assuming that we feel confident enough to start using this new process (Kyle and Mark might chime in on this point). Also, I would like to see this work completed by Juno-1, which I reckon is a reasonable target. Of course I'm available for discussing design, implementation, reviewing and writing code. Salvatore On 15 April 2014 12:44, Anna Kamyshnikova wrote: > Hello everyone! > > I would like to try to solve this problem. I registered blueprint on this > topic > https://blueprints.launchpad.net/neutron/+spec/neutron-robust-db-migrationsand > I'm going to experiment with options to solve this. I'm welcome any > suggestions and ready to talk about it via IRC (akamyshnikova). > > Regards > Ann > > > On Mon, Apr 14, 2014 at 9:39 PM, Sean Dague wrote: > >> On 04/14/2014 12:46 PM, Eugene Nikanorov wrote: >> > Hi Salvatore, >> > >> > The described problem could be even worse if vendor drivers are >> considered. >> > Doesn't #1 require that all DB tables are named differently? Otherwise >> > it seems that user can't be sure in DB schema even if all tables are >> > present. >> > >> > I think the big part of the problem is that we need to support both >> > online and offline migrations. Without the latter things could be a >> > little bit simpler. >> > >> > Also it seems to me that problem statement should be changed to the >> > following: >> > One need to migrate from (Config1, MigrationID1) to (Config2, >> > MigrationID2), and currently our code only accounts for MigrationIDs. >> > We may consider amending DB with configuration metadata, at least that >> > will allow to run migration code with full knowledge of what happened >> > before (if online mode is considered). >> > In offline mode that will require providing old and current >> configurations. >> > >> > That was just thinking aloud, no concrete proposals yet. >> >> The root issue really is Migrations *must* be global, and config >> invariant. That's the design point in both sqlalchemy-migrate and >> alembic. The fact that there is one global migrations table per >> database, with a single value in it, is indicative of this fact. >> >> I think that design point got lost somewhere along the way, and folks >> assumed migrations were just a way to change schemas. They are much more >> constrained than that. >> >> It does also sound like the data model is going to need some serious >> reconsidering given what's allowed to be changed at the plugin or vendor >> driver model. Contrast this with Nova, were virt drivers don't get to >> define persistant data that's unique to them (only generic data that >> they fit into the grander nova model). >> >> The one time we had a driver which needed persistent data (baremetal) it >> managed it's own database entirely. >> >> -Sean >> >> -- >> Sean Dague >> Samsung Research America >> s...@dague.net / sean.da...@samsung.com >> http://dague.net >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Neutron Migrations, service plugins and Grenade jobs
Hello everyone! I would like to try to solve this problem. I registered blueprint on this topic https://blueprints.launchpad.net/neutron/+spec/neutron-robust-db-migrationsand I'm going to experiment with options to solve this. I'm welcome any suggestions and ready to talk about it via IRC (akamyshnikova). Regards Ann On Mon, Apr 14, 2014 at 9:39 PM, Sean Dague wrote: > On 04/14/2014 12:46 PM, Eugene Nikanorov wrote: > > Hi Salvatore, > > > > The described problem could be even worse if vendor drivers are > considered. > > Doesn't #1 require that all DB tables are named differently? Otherwise > > it seems that user can't be sure in DB schema even if all tables are > > present. > > > > I think the big part of the problem is that we need to support both > > online and offline migrations. Without the latter things could be a > > little bit simpler. > > > > Also it seems to me that problem statement should be changed to the > > following: > > One need to migrate from (Config1, MigrationID1) to (Config2, > > MigrationID2), and currently our code only accounts for MigrationIDs. > > We may consider amending DB with configuration metadata, at least that > > will allow to run migration code with full knowledge of what happened > > before (if online mode is considered). > > In offline mode that will require providing old and current > configurations. > > > > That was just thinking aloud, no concrete proposals yet. > > The root issue really is Migrations *must* be global, and config > invariant. That's the design point in both sqlalchemy-migrate and > alembic. The fact that there is one global migrations table per > database, with a single value in it, is indicative of this fact. > > I think that design point got lost somewhere along the way, and folks > assumed migrations were just a way to change schemas. They are much more > constrained than that. > > It does also sound like the data model is going to need some serious > reconsidering given what's allowed to be changed at the plugin or vendor > driver model. Contrast this with Nova, were virt drivers don't get to > define persistant data that's unique to them (only generic data that > they fit into the grander nova model). > > The one time we had a driver which needed persistent data (baremetal) it > managed it's own database entirely. > > -Sean > > -- > Sean Dague > Samsung Research America > s...@dague.net / sean.da...@samsung.com > http://dague.net > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Neutron Migrations, service plugins and Grenade jobs
On 04/14/2014 12:46 PM, Eugene Nikanorov wrote: > Hi Salvatore, > > The described problem could be even worse if vendor drivers are considered. > Doesn't #1 require that all DB tables are named differently? Otherwise > it seems that user can't be sure in DB schema even if all tables are > present. > > I think the big part of the problem is that we need to support both > online and offline migrations. Without the latter things could be a > little bit simpler. > > Also it seems to me that problem statement should be changed to the > following: > One need to migrate from (Config1, MigrationID1) to (Config2, > MigrationID2), and currently our code only accounts for MigrationIDs. > We may consider amending DB with configuration metadata, at least that > will allow to run migration code with full knowledge of what happened > before (if online mode is considered). > In offline mode that will require providing old and current configurations. > > That was just thinking aloud, no concrete proposals yet. The root issue really is Migrations *must* be global, and config invariant. That's the design point in both sqlalchemy-migrate and alembic. The fact that there is one global migrations table per database, with a single value in it, is indicative of this fact. I think that design point got lost somewhere along the way, and folks assumed migrations were just a way to change schemas. They are much more constrained than that. It does also sound like the data model is going to need some serious reconsidering given what's allowed to be changed at the plugin or vendor driver model. Contrast this with Nova, were virt drivers don't get to define persistant data that's unique to them (only generic data that they fit into the grander nova model). The one time we had a driver which needed persistent data (baremetal) it managed it's own database entirely. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Migrations, service plugins and Grenade jobs
On 14 April 2014 17:27, Sean Dague wrote: > On 04/14/2014 12:09 PM, Kyle Mestery wrote: > > On Mon, Apr 14, 2014 at 10:54 AM, Salvatore Orlando > wrote: > > >>> The system could be made smarter by storing also a list of "known" > >>> migrations, including whether they were executed or skipped. > >>> > >>> Summarising, in my opinion the approach #2 is not even worth being > >>> considered. Between approach #1 and #3, I'd prefer the simplest one > and vote > >>> for approach #1. > >>> > > > > Thanks for sending out this summary email Salvatore. I agree with your > > summary and I think that option #1 is the way forward to fix this. The > > recovery scripts are a necessity here. I'd like to hear from some > > deployers who may be lurking on their thoughts on this approach as > > well. > > I think it's probably worth while to figure out how many branches > currently exist in the migrations path today? Basically how many > multiverses of neutron schemas might exist at the latest alembic id. > That will help inform how bad things might be. > To get this number, in theory, you should take in account every migration which could have been potentially skipped. Any of these migration is a potential "bisection" point of space-time continuum... and then you'll likely end up with a few tens of thousands of possible configurations. However, this will make things look just a lot worse then what they actually are. > > Grenade happened to trip over one of these, but that's just because of > which defaults we have. I expect that many more could be tripped over > and really burn people trying to make changes in their environment. > The areas where a fix is needed first are those related to "service plugins". So far load balancing, firewall, vpn, metering, and even l3 are treated as such. I will focus on rectifying migrations in these areas. Since migrations are executed conditionally, one thing to deal is that we don't know whether a given migration was executed or not, which makes things interesting, although not hard. > -Sean > > -- > Sean Dague > Samsung Research America > s...@dague.net / sean.da...@samsung.com > http://dague.net > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Neutron Migrations, service plugins and Grenade jobs
Hi Salvatore, The described problem could be even worse if vendor drivers are considered. Doesn't #1 require that all DB tables are named differently? Otherwise it seems that user can't be sure in DB schema even if all tables are present. I think the big part of the problem is that we need to support both online and offline migrations. Without the latter things could be a little bit simpler. Also it seems to me that problem statement should be changed to the following: One need to migrate from (Config1, MigrationID1) to (Config2, MigrationID2), and currently our code only accounts for MigrationIDs. We may consider amending DB with configuration metadata, at least that will allow to run migration code with full knowledge of what happened before (if online mode is considered). In offline mode that will require providing old and current configurations. That was just thinking aloud, no concrete proposals yet. Thanks, Eugene. On Mon, Apr 14, 2014 at 7:00 PM, Salvatore Orlando wrote: > This is a rather long post. However the gist of it is that Neutron > migrations are failing to correctly perform database upgrades when service > plugins are involved, and this probably means the conditional migration > path we designed for the Grizzly release is proving not robust enough when > dealing with service plugins. > > Corrective actions are therefore required; otherwise there is a risk the > Grenade jobs for neutron can't be made voting again. > This issue is already being tracked with bug 1307344 which was > independently filed and amended by Sean to reflect the actual issue. > > Grenade is currently failing because of this issue. Indeed the 'metering > plugin' is enabled in devstack for Icehouse but now Havana. > A quick fix for grenade has already been rejected (backport metering > support to stable/havana) another quick fix instead would be running > grenade without metering at all. > > However, whether we put or not a patch to make grenade running, there is > still an issue that needs to be solved. > Now, if you're not yet bored you might try now reading the long version of > this post... > > Problem statement: > > Enabling or disabling a service plugin in neutron configuration might be > done at any point and can lead migrations to build a database which is out > of sync with expectations > > Assume: > > Current migration X (Service plugin SP enabled here) > Service plugin SP introduced at migration F1 < X > No auto-generation of DB schemas > The migration F1 will be skipped > > When SP is enabled, the relevant DB schema won't exist and the feature > will not work. > > Other failure scenarios: > > Current migration X > SP introduced at migration F1 < X (but not yet enabled) > SP enabled at migration Y > F1 > X > SP updated at migration F2 | X < F2 < Y. > > F2 will fail because schemas are not yet ready for feature, even with auto > schema generation > > SP introduced at migration F0, and enabled at migration F0 > SP disabled at migration X > SP updated at migration F1, which is skipped because it's disabled > SP enabled again at migration Y. Feature won't work as migration X < F1 < > Y was never executed. > > Requirements for a possible solution: > > Should solve problem not only for greenfield deployments but also for > existing ones > Must not make migration generation more complex or introduce error-prone > behaviours > > Possible solutions: > > 1) Specify that all migrations must run for every plugin (*) unless they > are really introducing schemas which are specific to a particular > technology (such as uuid mappings between neutron and backed) > > This approach will probably ensure all the important migrations are > executed. > However, it is not back portable, unless deployers decide to tear down and > rebuild their databases from scratch, which is unlikely to happen. > > To this aim "recovery" scripts might be provided to sync up the schema > state of specific service plugins with the current alembic migration > registered in the neutron database; or appropriate migrations can be added > in the path to fix database schemas. > > (*) Neutron does not address, and probably never will address, switching > from plugin "a" to "b" for a given service (e.g.: core features). > > 2) Have plugin-specific migration paths. > > This is not supported by alembic. In this case probably the best solution > would be to have completely different databases for the main plugin and > service plugins, which is however not possible because of foreign key > relationships among tables in different service plugins. > > Also, there are likely to be migrations which alter core and service > schemas at the same time. Adapting these migrations might be a > problem.Extra work will also be needed to make this solution work with > existing deployments. > > 3) Make migrations smarter > > When migration X is executed, it checks whether any previous migration for > the plugins it alters has been executed. If not it will execute the mi
Re: [openstack-dev] [Neutron] Migrations, service plugins and Grenade jobs
On 04/14/2014 12:09 PM, Kyle Mestery wrote: > On Mon, Apr 14, 2014 at 10:54 AM, Salvatore Orlando > wrote: >>> The system could be made smarter by storing also a list of "known" >>> migrations, including whether they were executed or skipped. >>> >>> Summarising, in my opinion the approach #2 is not even worth being >>> considered. Between approach #1 and #3, I'd prefer the simplest one and vote >>> for approach #1. >>> > > Thanks for sending out this summary email Salvatore. I agree with your > summary and I think that option #1 is the way forward to fix this. The > recovery scripts are a necessity here. I'd like to hear from some > deployers who may be lurking on their thoughts on this approach as > well. I think it's probably worth while to figure out how many branches currently exist in the migrations path today? Basically how many multiverses of neutron schemas might exist at the latest alembic id. That will help inform how bad things might be. Grenade happened to trip over one of these, but that's just because of which defaults we have. I expect that many more could be tripped over and really burn people trying to make changes in their environment. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Migrations, service plugins and Grenade jobs
On Mon, Apr 14, 2014 at 10:54 AM, Salvatore Orlando wrote: > Resending with [Neutron] tag. > > Salvatore > > > On 14 April 2014 16:00, Salvatore Orlando wrote: >> >> This is a rather long post. However the gist of it is that Neutron >> migrations are failing to correctly perform database upgrades when service >> plugins are involved, and this probably means the conditional migration path >> we designed for the Grizzly release is proving not robust enough when >> dealing with service plugins. >> >> Corrective actions are therefore required; otherwise there is a risk the >> Grenade jobs for neutron can't be made voting again. >> This issue is already being tracked with bug 1307344 which was >> independently filed and amended by Sean to reflect the actual issue. >> >> Grenade is currently failing because of this issue. Indeed the 'metering >> plugin' is enabled in devstack for Icehouse but now Havana. >> A quick fix for grenade has already been rejected (backport metering >> support to stable/havana) another quick fix instead would be running grenade >> without metering at all. >> >> However, whether we put or not a patch to make grenade running, there is >> still an issue that needs to be solved. >> Now, if you're not yet bored you might try now reading the long version of >> this post... >> >> Problem statement: >> >> Enabling or disabling a service plugin in neutron configuration might be >> done at any point and can lead migrations to build a database which is out >> of sync with expectations >> >> Assume: >> >> Current migration X (Service plugin SP enabled here) >> Service plugin SP introduced at migration F1 < X >> No auto-generation of DB schemas >> The migration F1 will be skipped >> >> When SP is enabled, the relevant DB schema won't exist and the feature >> will not work. >> >> Other failure scenarios: >> >> Current migration X >> SP introduced at migration F1 < X (but not yet enabled) >> SP enabled at migration Y > F1 > X >> SP updated at migration F2 | X < F2 < Y. >> >> F2 will fail because schemas are not yet ready for feature, even with auto >> schema generation >> >> SP introduced at migration F0, and enabled at migration F0 >> SP disabled at migration X >> SP updated at migration F1, which is skipped because it's disabled >> SP enabled again at migration Y. Feature won't work as migration X < F1 < >> Y was never executed. >> >> Requirements for a possible solution: >> >> Should solve problem not only for greenfield deployments but also for >> existing ones >> Must not make migration generation more complex or introduce error-prone >> behaviours >> >> Possible solutions: >> >> 1) Specify that all migrations must run for every plugin (*) unless they >> are really introducing schemas which are specific to a particular technology >> (such as uuid mappings between neutron and backed) >> >> This approach will probably ensure all the important migrations are >> executed. >> However, it is not back portable, unless deployers decide to tear down and >> rebuild their databases from scratch, which is unlikely to happen. >> >> To this aim "recovery" scripts might be provided to sync up the schema >> state of specific service plugins with the current alembic migration >> registered in the neutron database; or appropriate migrations can be added >> in the path to fix database schemas. >> >> (*) Neutron does not address, and probably never will address, switching >> from plugin "a" to "b" for a given service (e.g.: core features). >> >> 2) Have plugin-specific migration paths. >> >> This is not supported by alembic. In this case probably the best solution >> would be to have completely different databases for the main plugin and >> service plugins, which is however not possible because of foreign key >> relationships among tables in different service plugins. >> >> Also, there are likely to be migrations which alter core and service >> schemas at the same time. Adapting these migrations might be a problem.Extra >> work will also be needed to make this solution work with existing >> deployments. >> >> 3) Make migrations smarter >> >> When migration X is executed, it checks whether any previous migration for >> the plugins it alters has been executed. If not it will execute the missing >> migrations as well. >> This *might* work; However, alembic currently offers no support for it, as >> it's not possible to perform any form of "introspection" in the migration >> themselves. >> >> The migration engine will also be augmented as the last executed migration >> for each plugin should be stored in the database. >> This might also prove to be error prone and lead to unexpected behaviours. >> For example: >> >> Until migration X deployment uses plugin A which provides services S1 and >> S2 >> At migration Y user still uses plugin for service S1, but switches to >> plugin B service S2 >> Migration Z>Y applies to plugin B. System detects no migration for plugin >> B have been executed so far. >> However, the migrati
Re: [openstack-dev] [Neutron] Migrations, service plugins and Grenade jobs
Resending with [Neutron] tag. Salvatore On 14 April 2014 16:00, Salvatore Orlando wrote: > This is a rather long post. However the gist of it is that Neutron > migrations are failing to correctly perform database upgrades when service > plugins are involved, and this probably means the conditional migration > path we designed for the Grizzly release is proving not robust enough when > dealing with service plugins. > > Corrective actions are therefore required; otherwise there is a risk the > Grenade jobs for neutron can't be made voting again. > This issue is already being tracked with bug 1307344 which was > independently filed and amended by Sean to reflect the actual issue. > > Grenade is currently failing because of this issue. Indeed the 'metering > plugin' is enabled in devstack for Icehouse but now Havana. > A quick fix for grenade has already been rejected (backport metering > support to stable/havana) another quick fix instead would be running > grenade without metering at all. > > However, whether we put or not a patch to make grenade running, there is > still an issue that needs to be solved. > Now, if you're not yet bored you might try now reading the long version of > this post... > > Problem statement: > > Enabling or disabling a service plugin in neutron configuration might be > done at any point and can lead migrations to build a database which is out > of sync with expectations > > Assume: > > Current migration X (Service plugin SP enabled here) > Service plugin SP introduced at migration F1 < X > No auto-generation of DB schemas > The migration F1 will be skipped > > When SP is enabled, the relevant DB schema won't exist and the feature > will not work. > > Other failure scenarios: > > Current migration X > SP introduced at migration F1 < X (but not yet enabled) > SP enabled at migration Y > F1 > X > SP updated at migration F2 | X < F2 < Y. > > F2 will fail because schemas are not yet ready for feature, even with auto > schema generation > > SP introduced at migration F0, and enabled at migration F0 > SP disabled at migration X > SP updated at migration F1, which is skipped because it's disabled > SP enabled again at migration Y. Feature won't work as migration X < F1 < > Y was never executed. > > Requirements for a possible solution: > > Should solve problem not only for greenfield deployments but also for > existing ones > Must not make migration generation more complex or introduce error-prone > behaviours > > Possible solutions: > > 1) Specify that all migrations must run for every plugin (*) unless they > are really introducing schemas which are specific to a particular > technology (such as uuid mappings between neutron and backed) > > This approach will probably ensure all the important migrations are > executed. > However, it is not back portable, unless deployers decide to tear down and > rebuild their databases from scratch, which is unlikely to happen. > > To this aim "recovery" scripts might be provided to sync up the schema > state of specific service plugins with the current alembic migration > registered in the neutron database; or appropriate migrations can be added > in the path to fix database schemas. > > (*) Neutron does not address, and probably never will address, switching > from plugin "a" to "b" for a given service (e.g.: core features). > > 2) Have plugin-specific migration paths. > > This is not supported by alembic. In this case probably the best solution > would be to have completely different databases for the main plugin and > service plugins, which is however not possible because of foreign key > relationships among tables in different service plugins. > > Also, there are likely to be migrations which alter core and service > schemas at the same time. Adapting these migrations might be a > problem.Extra work will also be needed to make this solution work with > existing deployments. > > 3) Make migrations smarter > > When migration X is executed, it checks whether any previous migration for > the plugins it alters has been executed. If not it will execute the missing > migrations as well. > This *might* work; However, alembic currently offers no support for it, as > it's not possible to perform any form of "introspection" in the migration > themselves. > > The migration engine will also be augmented as the last executed migration > for each plugin should be stored in the database. > This might also prove to be error prone and lead to unexpected behaviours. > For example: > > Until migration X deployment uses plugin A which provides services S1 and > S2 > At migration Y user still uses plugin for service S1, but switches to > plugin B service S2 > Migration Z>Y applies to plugin B. System detects no migration for plugin > B have been executed so far. > However, the migrations for the service S2, offered by B were already > executed when running plugin A > System tries to execute migrations for plugin B which will fail as they > were actually already exe
[openstack-dev] Neutron Migrations, service plugins and Grenade jobs
This is a rather long post. However the gist of it is that Neutron migrations are failing to correctly perform database upgrades when service plugins are involved, and this probably means the conditional migration path we designed for the Grizzly release is proving not robust enough when dealing with service plugins. Corrective actions are therefore required; otherwise there is a risk the Grenade jobs for neutron can't be made voting again. This issue is already being tracked with bug 1307344 which was independently filed and amended by Sean to reflect the actual issue. Grenade is currently failing because of this issue. Indeed the 'metering plugin' is enabled in devstack for Icehouse but now Havana. A quick fix for grenade has already been rejected (backport metering support to stable/havana) another quick fix instead would be running grenade without metering at all. However, whether we put or not a patch to make grenade running, there is still an issue that needs to be solved. Now, if you're not yet bored you might try now reading the long version of this post... Problem statement: Enabling or disabling a service plugin in neutron configuration might be done at any point and can lead migrations to build a database which is out of sync with expectations Assume: Current migration X (Service plugin SP enabled here) Service plugin SP introduced at migration F1 < X No auto-generation of DB schemas The migration F1 will be skipped When SP is enabled, the relevant DB schema won't exist and the feature will not work. Other failure scenarios: Current migration X SP introduced at migration F1 < X (but not yet enabled) SP enabled at migration Y > F1 > X SP updated at migration F2 | X < F2 < Y. F2 will fail because schemas are not yet ready for feature, even with auto schema generation SP introduced at migration F0, and enabled at migration F0 SP disabled at migration X SP updated at migration F1, which is skipped because it's disabled SP enabled again at migration Y. Feature won't work as migration X < F1 < Y was never executed. Requirements for a possible solution: Should solve problem not only for greenfield deployments but also for existing ones Must not make migration generation more complex or introduce error-prone behaviours Possible solutions: 1) Specify that all migrations must run for every plugin (*) unless they are really introducing schemas which are specific to a particular technology (such as uuid mappings between neutron and backed) This approach will probably ensure all the important migrations are executed. However, it is not back portable, unless deployers decide to tear down and rebuild their databases from scratch, which is unlikely to happen. To this aim "recovery" scripts might be provided to sync up the schema state of specific service plugins with the current alembic migration registered in the neutron database; or appropriate migrations can be added in the path to fix database schemas. (*) Neutron does not address, and probably never will address, switching from plugin "a" to "b" for a given service (e.g.: core features). 2) Have plugin-specific migration paths. This is not supported by alembic. In this case probably the best solution would be to have completely different databases for the main plugin and service plugins, which is however not possible because of foreign key relationships among tables in different service plugins. Also, there are likely to be migrations which alter core and service schemas at the same time. Adapting these migrations might be a problem.Extra work will also be needed to make this solution work with existing deployments. 3) Make migrations smarter When migration X is executed, it checks whether any previous migration for the plugins it alters has been executed. If not it will execute the missing migrations as well. This *might* work; However, alembic currently offers no support for it, as it's not possible to perform any form of "introspection" in the migration themselves. The migration engine will also be augmented as the last executed migration for each plugin should be stored in the database. This might also prove to be error prone and lead to unexpected behaviours. For example: Until migration X deployment uses plugin A which provides services S1 and S2 At migration Y user still uses plugin for service S1, but switches to plugin B service S2 Migration Z>Y applies to plugin B. System detects no migration for plugin B have been executed so far. However, the migrations for the service S2, offered by B were already executed when running plugin A System tries to execute migrations for plugin B which will fail as they were actually already executed for plugin A The system could be made smarter by storing also a list of "known" migrations, including whether they were executed or skipped. Summarising, in my opinion the approach #2 is not even worth being considered. Between approach #1 and #3, I'd prefer the simplest one and vote for app
