Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-02-02 Thread Adam Young

On 01/29/2015 03:11 PM, Mike Bayer wrote:


Morgan Fainberg  wrote:


Are downward migrations really a good idea for us to support? Is this downward 
migration path a sane expectation? In the real world, would any one really 
trust the data after migrating downwards?

It’s a good idea for a migration script to include a rudimentary downgrade 
operation to complement the upgrade operation, if feasible.  The purpose of 
this downgrade is from




Except that is it is code we need to maintain and support.  I think we 
are making more work for ourselves than the value these scripts provide 
justify.

  a practical standpoint helpful when locally testing a specific, typically 
small series of migrations.

A downgrade however typically only applies to schema objects, and not so much 
data.   It is often impossible to provide downgrades of data changes as it is 
likely that a data upgrade operation was destructive of some data.  Therefore, 
when dealing with a full series of real world migrations that include data 
migrations within them, downgrades are typically impossible.   I’m getting the 
impression that our migration scripts have data migrations galore in them.

So I am +1 on establishing a policy that the deployer of the application would 
not have access to any “downgrade” migrations, and -1 on removing “downgrade” 
entirely from individual migrations.   Specific migration scripts may return 
NotImplemented for their downgrade if its really not feasible, but for things 
like table and column changes where autogenerate has already rendered the 
downgrade, it’s handy to keep at least the smaller ones working.





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-02-02 Thread Adam Young

On 01/30/2015 07:23 AM, Sandy Walsh wrote:


From: Johannes Erdfelt [johan...@erdfelt.com]
Sent: Thursday, January 29, 2015 9:18 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

On Thu, Jan 29, 2015, Morgan Fainberg  wrote:

The concept that there is a utility that can (and in many cases
willfully) cause permanent, and in some cases irrevocable, data loss
from a simple command line interface sounds crazy when I try and
explain it to someone.

The more I work with the data stored in SQL, and the more I think we
should really recommend the tried-and-true best practices when trying
to revert from a migration: Restore your DB to a known good state.

You mean like restoring from backup?

Unless your code deploy fails before it has any chance of running, then
you could have had new instances started or instances changed and then
restoring from backups would lose data.

If you meant another way of restoring your data, then there are
some strategies that downgrades could employ that doesn't lose data,
but there is nothing that can handle 100% of cases.

All of that said, for the Rackspace Public Cloud, we have never rolled
back our deploy. We have always rolled forward for any fixes we needed.


>From my perspective, I'd be fine with doing away with downgrades, but

I'm not sure how to document that deployers should roll forward if they
have any deploy problems.

JE

Yep ... downgrades simply aren't practical with a SQL-schema based
solution. Too coarse-grained.

We'd have to move to a schema-less model, per-record versioning and
up-down conversion at the Nova Objects layer. Or, possibly introduce
more nodes that can deal with older versions. Either way, that's a big
hairy change


Horse pocky!  Schema less means "implied contract instead of implicit."  
That would be madness.  Please take the NoSQL good, SQL bad approach of 
of the conversation, as absotutely (yes, absotutely) everything we have 
here is doubly true for NoSQL, we just don't hammer on it as much.  We 
don't even document the record formats in the NoSQL cases in Keystone so 
we can break them both willy and nilly, but have often found that we are 
just stuck.  Usually, we are only dealing with the token table, and so 
we just dump the old tokens and shake our heads sadly.






.

The upgrade code is still required, so removing the downgrades (and
tests, if any) is a relatively small change to the code base.

The bigger issue is the anxiety the deployer will experience until a
patch lands.

-S

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-30 Thread Sandy Walsh

>
>From: Johannes Erdfelt [johan...@erdfelt.com]
>Sent: Thursday, January 29, 2015 9:18 PM
>To: OpenStack Development Mailing List (not for usage questions)
>Subject: Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues
>
>On Thu, Jan 29, 2015, Morgan Fainberg  wrote:
>> The concept that there is a utility that can (and in many cases
>> willfully) cause permanent, and in some cases irrevocable, data loss
>> from a simple command line interface sounds crazy when I try and
>> explain it to someone.
>>
>> The more I work with the data stored in SQL, and the more I think we
>> should really recommend the tried-and-true best practices when trying
>> to revert from a migration: Restore your DB to a known good state.
>
>You mean like restoring from backup?
>
>Unless your code deploy fails before it has any chance of running, then
>you could have had new instances started or instances changed and then
>restoring from backups would lose data.
>
>If you meant another way of restoring your data, then there are
>some strategies that downgrades could employ that doesn't lose data,
>but there is nothing that can handle 100% of cases.
>
>All of that said, for the Rackspace Public Cloud, we have never rolled
>back our deploy. We have always rolled forward for any fixes we needed.
>
>From my perspective, I'd be fine with doing away with downgrades, but
>I'm not sure how to document that deployers should roll forward if they
>have any deploy problems.
>
>JE

Yep ... downgrades simply aren't practical with a SQL-schema based
solution. Too coarse-grained.

We'd have to move to a schema-less model, per-record versioning and
up-down conversion at the Nova Objects layer. Or, possibly introduce
more nodes that can deal with older versions. Either way, that's a big
hairy change.

The upgrade code is still required, so removing the downgrades (and
tests, if any) is a relatively small change to the code base.

The bigger issue is the anxiety the deployer will experience until a
patch lands.

-S

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-30 Thread Boris Bobrov
On Friday 30 January 2015 01:01:00 Boris Bobrov wrote:
> On Thursday 29 January 2015 22:06:25 Morgan Fainberg wrote:
> > I’d like to propose we stop setting the expectation that a downwards
> > migration is a “good idea” or even something we should really support.
> > Offering upwards-only migrations would also simplify the migrations in
> > general. This downward migration path is also somewhat broken by the
> > migration collapses performed in a number of projects (to limit the
> > number of migrations that need to be updated when we change a key
> > component such as oslo.db or SQL-Alchemy Migrate to Alembic).
> > 
> > Are downward migrations really a good idea for us to support? Is this
> > downward migration path a sane expectation? In the real world, would any
> > one really trust the data after migrating downwards?
> 
> Frankly, I don't see a case when a downgrade from n to (n - 1) in
> development cannot be replaced with a set of fixtures and upgrade from 0
> to (n - 1).
> 
> If we assume that upgrade can possible break something in production, we
> should not rely on fixing by downgrading the schema, because a) the code
> depends on the latest schema and b) "break" can be different and
> unrecoverable.
> 
> IMO downward migrations should be disabled. We could make a survey though,
> maybe someone has a story of using them in the fields.

I've made a little survey and there are people who used downgrades for 
debugging of different OpenStack releases.

So, I think I'm +1 on Mike Bayer's opinion.

-- 
Best regards,
Boris Bobrov

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-29 Thread Johannes Erdfelt
On Thu, Jan 29, 2015, Morgan Fainberg  wrote:
> The concept that there is a utility that can (and in many cases
> willfully) cause permanent, and in some cases irrevocable, data loss
> from a simple command line interface sounds crazy when I try and
> explain it to someone.
> 
> The more I work with the data stored in SQL, and the more I think we
> should really recommend the tried-and-true best practices when trying
> to revert from a migration: Restore your DB to a known good state.

You mean like restoring from backup?

Unless your code deploy fails before it has any chance of running, then
you could have had new instances started or instances changed and then
restoring from backups would lose data.

If you meant another way of restoring your data, then there are
some strategies that downgrades could employ that doesn't lose data,
but there is nothing that can handle 100% of cases.

All of that said, for the Rackspace Public Cloud, we have never rolled
back our deploy. We have always rolled forward for any fixes we needed.

>From my perspective, I'd be fine with doing away with downgrades, but
I'm not sure how to document that deployers should roll forward if they
have any deploy problems.

JE


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-29 Thread Boris Bobrov
On Thursday 29 January 2015 22:06:25 Morgan Fainberg wrote:
> I’d like to propose we stop setting the expectation that a downwards
> migration is a “good idea” or even something we should really support.
> Offering upwards-only migrations would also simplify the migrations in
> general. This downward migration path is also somewhat broken by the
> migration collapses performed in a number of projects (to limit the number
> of migrations that need to be updated when we change a key component such
> as oslo.db or SQL-Alchemy Migrate to Alembic).
> 
> Are downward migrations really a good idea for us to support? Is this
> downward migration path a sane expectation? In the real world, would any
> one really trust the data after migrating downwards?

Frankly, I don't see a case when a downgrade from n to (n - 1) in development 
cannot be replaced with a set of fixtures and upgrade from 0 to (n - 1).

If we assume that upgrade can possible break something in production, we 
should not rely on fixing by downgrading the schema, because a) the code 
depends on the latest schema and b) "break" can be different and unrecoverable.

IMO downward migrations should be disabled. We could make a survey though, 
maybe someone has a story of using them in the fields.

-- 
С наилучшими пожеланиями,
Boris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-29 Thread Mike Bayer


Morgan Fainberg  wrote:

> 
> Are downward migrations really a good idea for us to support? Is this 
> downward migration path a sane expectation? In the real world, would any one 
> really trust the data after migrating downwards?

It’s a good idea for a migration script to include a rudimentary downgrade 
operation to complement the upgrade operation, if feasible.  The purpose of 
this downgrade is from a practical standpoint helpful when locally testing a 
specific, typically small series of migrations.

A downgrade however typically only applies to schema objects, and not so much 
data.   It is often impossible to provide downgrades of data changes as it is 
likely that a data upgrade operation was destructive of some data.  Therefore, 
when dealing with a full series of real world migrations that include data 
migrations within them, downgrades are typically impossible.   I’m getting the 
impression that our migration scripts have data migrations galore in them.   

So I am +1 on establishing a policy that the deployer of the application would 
not have access to any “downgrade” migrations, and -1 on removing “downgrade” 
entirely from individual migrations.   Specific migration scripts may return 
NotImplemented for their downgrade if its really not feasible, but for things 
like table and column changes where autogenerate has already rendered the 
downgrade, it’s handy to keep at least the smaller ones working.





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-29 Thread gordon chung



> >> Are downward migrations really a good idea for us to support? Is this 
> >> downward migration path a sane expectation? In the real world, would any 
> >> one really trust the data after migrating downwards?
are downward migrations actually mandated? i always assumed it was just been a 
pain that no one bother tried breaking. i'd be all for ending the downward path 
-- i would assume/hope most people upgrading have some sort of fail back 
(backup) already.
cheers,
gord
  __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-29 Thread Donald Stufft

> On Jan 29, 2015, at 2:27 PM, Monty Taylor  wrote:
> 
> On 01/29/2015 11:06 AM, Morgan Fainberg wrote:
>> As a quick preface, today there is the assumption you can upgrade and 
>> downgrade your SQL Schema. For the most part we do our best to test all of 
>> this in our unit tests (do upgrades and downgrades land us in the same 
>> schema). What isn’t clearly addressed is that the concept of downgrade might 
>> be inherently flawed, especially when you’re talking about the data 
>> associated with the migration. The concept that there is a utility that can 
>> (and in many cases willfully) cause permanent, and in some cases 
>> irrevocable, data loss from a simple command line interface sounds crazy 
>> when I try and explain it to someone.
>> 
>> The more I work with the data stored in SQL, and the more I think we should 
>> really recommend the tried-and-true best practices when trying to revert 
>> from a migration: Restore your DB to a known good state.
>> 
>> * If a migration fails in some spectacular way (that was not handled 
>> gracefully) is it possible to use the downgrade mechanic to “fix” it? More 
>> importantly, would you trust the data after that downgrade?
>> * Once an upgrade has happened and new code is run (potentially making use 
>> of the new data structures), is it really safe to downgrade and lose that 
>> data without stepping back to a known consistent state?
>> 
>> The other side of this coin is that it prevents us from collapsing data 
>> types in the stable store without hints to reverse the migration.
>> 
>> I get the feeling that the reason we provide downward migrations today is 
>> because we have in the past. Due to the existence of these downward 
>> migrations the expectation is that they will work, and so we’re in a weird 
>> feedback-loop.
>> 
>> I’d like to propose we stop setting the expectation that a downwards 
>> migration is a “good idea” or even something we should really support. 
>> Offering upwards-only migrations would also simplify the migrations in 
>> general. This downward migration path is also somewhat broken by the 
>> migration collapses performed in a number of projects (to limit the number 
>> of migrations that need to be updated when we change a key component such as 
>> oslo.db or SQL-Alchemy Migrate to Alembic).
>> 
>> Are downward migrations really a good idea for us to support? Is this 
>> downward migration path a sane expectation? In the real world, would any one 
>> really trust the data after migrating downwards?
> 
> I do not think downward migrations are a good idea. I think they are a
> spectacularly bad idea that is essentially designed to get users into a
> state where they are massively broken.
> 
> Operators should fail forward or restore from backup. Giving them
> downgrade scripts will imply that they work, which they probably will
> not once actual data is involved.

+1 on disabling downgrades.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-29 Thread Monty Taylor
On 01/29/2015 11:06 AM, Morgan Fainberg wrote:
> As a quick preface, today there is the assumption you can upgrade and 
> downgrade your SQL Schema. For the most part we do our best to test all of 
> this in our unit tests (do upgrades and downgrades land us in the same 
> schema). What isn’t clearly addressed is that the concept of downgrade might 
> be inherently flawed, especially when you’re talking about the data 
> associated with the migration. The concept that there is a utility that can 
> (and in many cases willfully) cause permanent, and in some cases irrevocable, 
> data loss from a simple command line interface sounds crazy when I try and 
> explain it to someone.
> 
> The more I work with the data stored in SQL, and the more I think we should 
> really recommend the tried-and-true best practices when trying to revert from 
> a migration: Restore your DB to a known good state.
> 
> * If a migration fails in some spectacular way (that was not handled 
> gracefully) is it possible to use the downgrade mechanic to “fix” it? More 
> importantly, would you trust the data after that downgrade?
> * Once an upgrade has happened and new code is run (potentially making use of 
> the new data structures), is it really safe to downgrade and lose that data 
> without stepping back to a known consistent state?
> 
> The other side of this coin is that it prevents us from collapsing data types 
> in the stable store without hints to reverse the migration.
> 
> I get the feeling that the reason we provide downward migrations today is 
> because we have in the past. Due to the existence of these downward 
> migrations the expectation is that they will work, and so we’re in a weird 
> feedback-loop.
> 
> I’d like to propose we stop setting the expectation that a downwards 
> migration is a “good idea” or even something we should really support. 
> Offering upwards-only migrations would also simplify the migrations in 
> general. This downward migration path is also somewhat broken by the 
> migration collapses performed in a number of projects (to limit the number of 
> migrations that need to be updated when we change a key component such as 
> oslo.db or SQL-Alchemy Migrate to Alembic).
> 
> Are downward migrations really a good idea for us to support? Is this 
> downward migration path a sane expectation? In the real world, would any one 
> really trust the data after migrating downwards?

I do not think downward migrations are a good idea. I think they are a
spectacularly bad idea that is essentially designed to get users into a
state where they are massively broken.

Operators should fail forward or restore from backup. Giving them
downgrade scripts will imply that they work, which they probably will
not once actual data is involved.

Monty


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-29 Thread Morgan Fainberg
As a quick preface, today there is the assumption you can upgrade and downgrade 
your SQL Schema. For the most part we do our best to test all of this in our 
unit tests (do upgrades and downgrades land us in the same schema). What isn’t 
clearly addressed is that the concept of downgrade might be inherently flawed, 
especially when you’re talking about the data associated with the migration. 
The concept that there is a utility that can (and in many cases willfully) 
cause permanent, and in some cases irrevocable, data loss from a simple command 
line interface sounds crazy when I try and explain it to someone.

The more I work with the data stored in SQL, and the more I think we should 
really recommend the tried-and-true best practices when trying to revert from a 
migration: Restore your DB to a known good state.

* If a migration fails in some spectacular way (that was not handled 
gracefully) is it possible to use the downgrade mechanic to “fix” it? More 
importantly, would you trust the data after that downgrade?
* Once an upgrade has happened and new code is run (potentially making use of 
the new data structures), is it really safe to downgrade and lose that data 
without stepping back to a known consistent state?

The other side of this coin is that it prevents us from collapsing data types 
in the stable store without hints to reverse the migration.

I get the feeling that the reason we provide downward migrations today is 
because we have in the past. Due to the existence of these downward migrations 
the expectation is that they will work, and so we’re in a weird feedback-loop.

I’d like to propose we stop setting the expectation that a downwards migration 
is a “good idea” or even something we should really support. Offering 
upwards-only migrations would also simplify the migrations in general. This 
downward migration path is also somewhat broken by the migration collapses 
performed in a number of projects (to limit the number of migrations that need 
to be updated when we change a key component such as oslo.db or SQL-Alchemy 
Migrate to Alembic).

Are downward migrations really a good idea for us to support? Is this downward 
migration path a sane expectation? In the real world, would any one really 
trust the data after migrating downwards?

Thanks for reading!
—Morgan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev