Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
On 01/29/2015 03:11 PM, Mike Bayer wrote: Morgan Fainberg wrote: Are downward migrations really a good idea for us to support? Is this downward migration path a sane expectation? In the real world, would any one really trust the data after migrating downwards? It’s a good idea for a migration script to include a rudimentary downgrade operation to complement the upgrade operation, if feasible. The purpose of this downgrade is from Except that is it is code we need to maintain and support. I think we are making more work for ourselves than the value these scripts provide justify. a practical standpoint helpful when locally testing a specific, typically small series of migrations. A downgrade however typically only applies to schema objects, and not so much data. It is often impossible to provide downgrades of data changes as it is likely that a data upgrade operation was destructive of some data. Therefore, when dealing with a full series of real world migrations that include data migrations within them, downgrades are typically impossible. I’m getting the impression that our migration scripts have data migrations galore in them. So I am +1 on establishing a policy that the deployer of the application would not have access to any “downgrade” migrations, and -1 on removing “downgrade” entirely from individual migrations. Specific migration scripts may return NotImplemented for their downgrade if its really not feasible, but for things like table and column changes where autogenerate has already rendered the downgrade, it’s handy to keep at least the smaller ones working. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
On 01/30/2015 07:23 AM, Sandy Walsh wrote: From: Johannes Erdfelt [johan...@erdfelt.com] Sent: Thursday, January 29, 2015 9:18 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues On Thu, Jan 29, 2015, Morgan Fainberg wrote: The concept that there is a utility that can (and in many cases willfully) cause permanent, and in some cases irrevocable, data loss from a simple command line interface sounds crazy when I try and explain it to someone. The more I work with the data stored in SQL, and the more I think we should really recommend the tried-and-true best practices when trying to revert from a migration: Restore your DB to a known good state. You mean like restoring from backup? Unless your code deploy fails before it has any chance of running, then you could have had new instances started or instances changed and then restoring from backups would lose data. If you meant another way of restoring your data, then there are some strategies that downgrades could employ that doesn't lose data, but there is nothing that can handle 100% of cases. All of that said, for the Rackspace Public Cloud, we have never rolled back our deploy. We have always rolled forward for any fixes we needed. >From my perspective, I'd be fine with doing away with downgrades, but I'm not sure how to document that deployers should roll forward if they have any deploy problems. JE Yep ... downgrades simply aren't practical with a SQL-schema based solution. Too coarse-grained. We'd have to move to a schema-less model, per-record versioning and up-down conversion at the Nova Objects layer. Or, possibly introduce more nodes that can deal with older versions. Either way, that's a big hairy change Horse pocky! Schema less means "implied contract instead of implicit." That would be madness. Please take the NoSQL good, SQL bad approach of of the conversation, as absotutely (yes, absotutely) everything we have here is doubly true for NoSQL, we just don't hammer on it as much. We don't even document the record formats in the NoSQL cases in Keystone so we can break them both willy and nilly, but have often found that we are just stuck. Usually, we are only dealing with the token table, and so we just dump the old tokens and shake our heads sadly. . The upgrade code is still required, so removing the downgrades (and tests, if any) is a relatively small change to the code base. The bigger issue is the anxiety the deployer will experience until a patch lands. -S __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
> >From: Johannes Erdfelt [johan...@erdfelt.com] >Sent: Thursday, January 29, 2015 9:18 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues > >On Thu, Jan 29, 2015, Morgan Fainberg wrote: >> The concept that there is a utility that can (and in many cases >> willfully) cause permanent, and in some cases irrevocable, data loss >> from a simple command line interface sounds crazy when I try and >> explain it to someone. >> >> The more I work with the data stored in SQL, and the more I think we >> should really recommend the tried-and-true best practices when trying >> to revert from a migration: Restore your DB to a known good state. > >You mean like restoring from backup? > >Unless your code deploy fails before it has any chance of running, then >you could have had new instances started or instances changed and then >restoring from backups would lose data. > >If you meant another way of restoring your data, then there are >some strategies that downgrades could employ that doesn't lose data, >but there is nothing that can handle 100% of cases. > >All of that said, for the Rackspace Public Cloud, we have never rolled >back our deploy. We have always rolled forward for any fixes we needed. > >From my perspective, I'd be fine with doing away with downgrades, but >I'm not sure how to document that deployers should roll forward if they >have any deploy problems. > >JE Yep ... downgrades simply aren't practical with a SQL-schema based solution. Too coarse-grained. We'd have to move to a schema-less model, per-record versioning and up-down conversion at the Nova Objects layer. Or, possibly introduce more nodes that can deal with older versions. Either way, that's a big hairy change. The upgrade code is still required, so removing the downgrades (and tests, if any) is a relatively small change to the code base. The bigger issue is the anxiety the deployer will experience until a patch lands. -S __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
On Friday 30 January 2015 01:01:00 Boris Bobrov wrote: > On Thursday 29 January 2015 22:06:25 Morgan Fainberg wrote: > > I’d like to propose we stop setting the expectation that a downwards > > migration is a “good idea” or even something we should really support. > > Offering upwards-only migrations would also simplify the migrations in > > general. This downward migration path is also somewhat broken by the > > migration collapses performed in a number of projects (to limit the > > number of migrations that need to be updated when we change a key > > component such as oslo.db or SQL-Alchemy Migrate to Alembic). > > > > Are downward migrations really a good idea for us to support? Is this > > downward migration path a sane expectation? In the real world, would any > > one really trust the data after migrating downwards? > > Frankly, I don't see a case when a downgrade from n to (n - 1) in > development cannot be replaced with a set of fixtures and upgrade from 0 > to (n - 1). > > If we assume that upgrade can possible break something in production, we > should not rely on fixing by downgrading the schema, because a) the code > depends on the latest schema and b) "break" can be different and > unrecoverable. > > IMO downward migrations should be disabled. We could make a survey though, > maybe someone has a story of using them in the fields. I've made a little survey and there are people who used downgrades for debugging of different OpenStack releases. So, I think I'm +1 on Mike Bayer's opinion. -- Best regards, Boris Bobrov __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
On Thu, Jan 29, 2015, Morgan Fainberg wrote: > The concept that there is a utility that can (and in many cases > willfully) cause permanent, and in some cases irrevocable, data loss > from a simple command line interface sounds crazy when I try and > explain it to someone. > > The more I work with the data stored in SQL, and the more I think we > should really recommend the tried-and-true best practices when trying > to revert from a migration: Restore your DB to a known good state. You mean like restoring from backup? Unless your code deploy fails before it has any chance of running, then you could have had new instances started or instances changed and then restoring from backups would lose data. If you meant another way of restoring your data, then there are some strategies that downgrades could employ that doesn't lose data, but there is nothing that can handle 100% of cases. All of that said, for the Rackspace Public Cloud, we have never rolled back our deploy. We have always rolled forward for any fixes we needed. >From my perspective, I'd be fine with doing away with downgrades, but I'm not sure how to document that deployers should roll forward if they have any deploy problems. JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
On Thursday 29 January 2015 22:06:25 Morgan Fainberg wrote: > I’d like to propose we stop setting the expectation that a downwards > migration is a “good idea” or even something we should really support. > Offering upwards-only migrations would also simplify the migrations in > general. This downward migration path is also somewhat broken by the > migration collapses performed in a number of projects (to limit the number > of migrations that need to be updated when we change a key component such > as oslo.db or SQL-Alchemy Migrate to Alembic). > > Are downward migrations really a good idea for us to support? Is this > downward migration path a sane expectation? In the real world, would any > one really trust the data after migrating downwards? Frankly, I don't see a case when a downgrade from n to (n - 1) in development cannot be replaced with a set of fixtures and upgrade from 0 to (n - 1). If we assume that upgrade can possible break something in production, we should not rely on fixing by downgrading the schema, because a) the code depends on the latest schema and b) "break" can be different and unrecoverable. IMO downward migrations should be disabled. We could make a survey though, maybe someone has a story of using them in the fields. -- С наилучшими пожеланиями, Boris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
Morgan Fainberg wrote: > > Are downward migrations really a good idea for us to support? Is this > downward migration path a sane expectation? In the real world, would any one > really trust the data after migrating downwards? It’s a good idea for a migration script to include a rudimentary downgrade operation to complement the upgrade operation, if feasible. The purpose of this downgrade is from a practical standpoint helpful when locally testing a specific, typically small series of migrations. A downgrade however typically only applies to schema objects, and not so much data. It is often impossible to provide downgrades of data changes as it is likely that a data upgrade operation was destructive of some data. Therefore, when dealing with a full series of real world migrations that include data migrations within them, downgrades are typically impossible. I’m getting the impression that our migration scripts have data migrations galore in them. So I am +1 on establishing a policy that the deployer of the application would not have access to any “downgrade” migrations, and -1 on removing “downgrade” entirely from individual migrations. Specific migration scripts may return NotImplemented for their downgrade if its really not feasible, but for things like table and column changes where autogenerate has already rendered the downgrade, it’s handy to keep at least the smaller ones working. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
> >> Are downward migrations really a good idea for us to support? Is this > >> downward migration path a sane expectation? In the real world, would any > >> one really trust the data after migrating downwards? are downward migrations actually mandated? i always assumed it was just been a pain that no one bother tried breaking. i'd be all for ending the downward path -- i would assume/hope most people upgrading have some sort of fail back (backup) already. cheers, gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
> On Jan 29, 2015, at 2:27 PM, Monty Taylor wrote: > > On 01/29/2015 11:06 AM, Morgan Fainberg wrote: >> As a quick preface, today there is the assumption you can upgrade and >> downgrade your SQL Schema. For the most part we do our best to test all of >> this in our unit tests (do upgrades and downgrades land us in the same >> schema). What isn’t clearly addressed is that the concept of downgrade might >> be inherently flawed, especially when you’re talking about the data >> associated with the migration. The concept that there is a utility that can >> (and in many cases willfully) cause permanent, and in some cases >> irrevocable, data loss from a simple command line interface sounds crazy >> when I try and explain it to someone. >> >> The more I work with the data stored in SQL, and the more I think we should >> really recommend the tried-and-true best practices when trying to revert >> from a migration: Restore your DB to a known good state. >> >> * If a migration fails in some spectacular way (that was not handled >> gracefully) is it possible to use the downgrade mechanic to “fix” it? More >> importantly, would you trust the data after that downgrade? >> * Once an upgrade has happened and new code is run (potentially making use >> of the new data structures), is it really safe to downgrade and lose that >> data without stepping back to a known consistent state? >> >> The other side of this coin is that it prevents us from collapsing data >> types in the stable store without hints to reverse the migration. >> >> I get the feeling that the reason we provide downward migrations today is >> because we have in the past. Due to the existence of these downward >> migrations the expectation is that they will work, and so we’re in a weird >> feedback-loop. >> >> I’d like to propose we stop setting the expectation that a downwards >> migration is a “good idea” or even something we should really support. >> Offering upwards-only migrations would also simplify the migrations in >> general. This downward migration path is also somewhat broken by the >> migration collapses performed in a number of projects (to limit the number >> of migrations that need to be updated when we change a key component such as >> oslo.db or SQL-Alchemy Migrate to Alembic). >> >> Are downward migrations really a good idea for us to support? Is this >> downward migration path a sane expectation? In the real world, would any one >> really trust the data after migrating downwards? > > I do not think downward migrations are a good idea. I think they are a > spectacularly bad idea that is essentially designed to get users into a > state where they are massively broken. > > Operators should fail forward or restore from backup. Giving them > downgrade scripts will imply that they work, which they probably will > not once actual data is involved. +1 on disabling downgrades. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
On 01/29/2015 11:06 AM, Morgan Fainberg wrote: > As a quick preface, today there is the assumption you can upgrade and > downgrade your SQL Schema. For the most part we do our best to test all of > this in our unit tests (do upgrades and downgrades land us in the same > schema). What isn’t clearly addressed is that the concept of downgrade might > be inherently flawed, especially when you’re talking about the data > associated with the migration. The concept that there is a utility that can > (and in many cases willfully) cause permanent, and in some cases irrevocable, > data loss from a simple command line interface sounds crazy when I try and > explain it to someone. > > The more I work with the data stored in SQL, and the more I think we should > really recommend the tried-and-true best practices when trying to revert from > a migration: Restore your DB to a known good state. > > * If a migration fails in some spectacular way (that was not handled > gracefully) is it possible to use the downgrade mechanic to “fix” it? More > importantly, would you trust the data after that downgrade? > * Once an upgrade has happened and new code is run (potentially making use of > the new data structures), is it really safe to downgrade and lose that data > without stepping back to a known consistent state? > > The other side of this coin is that it prevents us from collapsing data types > in the stable store without hints to reverse the migration. > > I get the feeling that the reason we provide downward migrations today is > because we have in the past. Due to the existence of these downward > migrations the expectation is that they will work, and so we’re in a weird > feedback-loop. > > I’d like to propose we stop setting the expectation that a downwards > migration is a “good idea” or even something we should really support. > Offering upwards-only migrations would also simplify the migrations in > general. This downward migration path is also somewhat broken by the > migration collapses performed in a number of projects (to limit the number of > migrations that need to be updated when we change a key component such as > oslo.db or SQL-Alchemy Migrate to Alembic). > > Are downward migrations really a good idea for us to support? Is this > downward migration path a sane expectation? In the real world, would any one > really trust the data after migrating downwards? I do not think downward migrations are a good idea. I think they are a spectacularly bad idea that is essentially designed to get users into a state where they are massively broken. Operators should fail forward or restore from backup. Giving them downgrade scripts will imply that they work, which they probably will not once actual data is involved. Monty __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
As a quick preface, today there is the assumption you can upgrade and downgrade your SQL Schema. For the most part we do our best to test all of this in our unit tests (do upgrades and downgrades land us in the same schema). What isn’t clearly addressed is that the concept of downgrade might be inherently flawed, especially when you’re talking about the data associated with the migration. The concept that there is a utility that can (and in many cases willfully) cause permanent, and in some cases irrevocable, data loss from a simple command line interface sounds crazy when I try and explain it to someone. The more I work with the data stored in SQL, and the more I think we should really recommend the tried-and-true best practices when trying to revert from a migration: Restore your DB to a known good state. * If a migration fails in some spectacular way (that was not handled gracefully) is it possible to use the downgrade mechanic to “fix” it? More importantly, would you trust the data after that downgrade? * Once an upgrade has happened and new code is run (potentially making use of the new data structures), is it really safe to downgrade and lose that data without stepping back to a known consistent state? The other side of this coin is that it prevents us from collapsing data types in the stable store without hints to reverse the migration. I get the feeling that the reason we provide downward migrations today is because we have in the past. Due to the existence of these downward migrations the expectation is that they will work, and so we’re in a weird feedback-loop. I’d like to propose we stop setting the expectation that a downwards migration is a “good idea” or even something we should really support. Offering upwards-only migrations would also simplify the migrations in general. This downward migration path is also somewhat broken by the migration collapses performed in a number of projects (to limit the number of migrations that need to be updated when we change a key component such as oslo.db or SQL-Alchemy Migrate to Alembic). Are downward migrations really a good idea for us to support? Is this downward migration path a sane expectation? In the real world, would any one really trust the data after migrating downwards? Thanks for reading! —Morgan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
