Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Tim Bell]

Glance provides a very nice set up for this

- Default is no delayed deletion
- Length of time before scrubbing is configurable
- The clean up process is automated using the glance scrubber which can be run 
as a standalone job or as a daemon

Tim

> -Original Message-
> From: Radomir Dopieralski [mailto:openst...@sheep.art.pl]
> Sent: 17 March 2014 10:33
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft 
> deletion (step by step)
> 
> On 16/03/14 06:04, Clint Byrum wrote:
> 
> > I think you can achieve this level of protection simply by denying
> > interactive users the rights to delete individual things directly, and
> > using stop instead of delete. Then have something else (cron?) clean
> > up stopped instances after a safety period has been reached.
> 
> I would be very interested in the approach to determining the optimal value 
> for that safety period you are mentioning. Or is this
> going to be left as an exercise for the reader? (That is, set in the 
> configuration, so that the users have to somehow perform this
> impossible task.)
> 
> --
> Radomir Dopieralski
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Radomir Dopieralski]

On 16/03/14 06:04, Clint Byrum wrote:

> I think you can achieve this level of protection simply by denying
> interactive users the rights to delete individual things directly, and
> using stop instead of delete. Then have something else (cron?) clean up
> stopped instances after a safety period has been reached.

I would be very interested in the approach to determining the optimal
value for that safety period you are mentioning. Or is this going to be
left as an exercise for the reader? (That is, set in the configuration,
so that the users have to somehow perform this impossible task.)

-- 
Radomir Dopieralski


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Clint Byrum]

Excerpts from Tim Bell's message of 2014-03-14 13:54:32 -0700:
> 
> I think we need to split the scenarios and focus on the end user experience 
> with the cloud 
> 
>  a few come to my mind from the CERN experience (but this may not be all):
> 
> 1. Accidental deletion of an object (including meta data)
> 2. Multi-level consistency (such as between Cell API and child instances)
> 3. Auditing
> 
> CERN has the scenario 1 at a reasonable frequency. Ultimately, it is due to 
> error by
> --
> A - the openstack administrators themselves
> B - the delegated project administrators
> C - users with a non-optimised scope for administrative action
> D - users who make mistakes
> 
> It seems that we should handle these as different cases
> 
> 3 - make sure there is a log entry (ideally off the box) for all operations
> 2 - up to the component implementers but with the aim to expire deleted 
> entries as soon as reasonable consistency is achieved
> 1[A-D] - how can we recover from operator/project admin/user error ?
> 
> I understand that there are differing perspectives from cloud to server 
> consolidation but my cloud users expect that if they create a one-off virtual 
> desktop running Windows for software testing and install a set of software, I 
> don't tell them it was accidentally deleted due to operator error (1A or 1B), 
> you need to re-create it.
> 


Totally agree with all of your points.

I think you can achieve this level of protection simply by denying
interactive users the rights to delete individual things directly, and
using stop instead of delete. Then have something else (cron?) clean up
stopped instances after a safety period has been reached.

This is an interesting counter to the opposite more fluid tactic which
is to delete instances that have been up for too long, assuming that
long lived == wrong and costly.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Tim Bell]

I think we need to split the scenarios and focus on the end user experience 
with the cloud 

 a few come to my mind from the CERN experience (but this may not be all):

1. Accidental deletion of an object (including meta data)
2. Multi-level consistency (such as between Cell API and child instances)
3. Auditing

CERN has the scenario 1 at a reasonable frequency. Ultimately, it is due to 
error by
--
A - the openstack administrators themselves
B - the delegated project administrators
C - users with a non-optimised scope for administrative action
D - users who make mistakes

It seems that we should handle these as different cases

3 - make sure there is a log entry (ideally off the box) for all operations
2 - up to the component implementers but with the aim to expire deleted entries 
as soon as reasonable consistency is achieved
1[A-D] - how can we recover from operator/project admin/user error ?

I understand that there are differing perspectives from cloud to server 
consolidation but my cloud users expect that if they create a one-off virtual 
desktop running Windows for software testing and install a set of software, I 
don't tell them it was accidentally deleted due to operator error (1A or 1B), 
you need to re-create it.

Tim

> -Original Message-
> From: Jay Pipes [mailto:jaypi...@gmail.com]
> Sent: 14 March 2014 16:55
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft 
> deletion (step by step)
> 
> On Fri, 2014-03-14 at 08:37 +0100, Radomir Dopieralski wrote:
> > Hello,
> >
> > I also think that this thread is going in the wrong direction, but I
> > don't think the direction Boris wants is the correct one either.
> > Frankly I'm a little surprised that nobody mentioned another advantage
> > that soft delete gives us, the one that I think it was actually used for 
> > originally.
> >
> > You see, soft delete is an optimization. It's there to make the system
> > work faster as a whole, have less code and be simpler to maintain and debug.
> >
> > How does it do it, when, as clearly shown in the first post in this
> > thread, it makes the queries slower, requires additional indices in
> > the database and more logic in the queries?
> 
> I feel it isn't an optimization if:
> 
> * It slows down the code base
> * Makes the code harder to read and understand
> * Deliberately obscures the actions of removing and restoring resources
> * Encourages the idea that everything in the system is "undoable", like the 
> cloud is a Word doc.
> 
> >  The answer is, by doing more
> > with those queries, by making you write less code, execute fewer
> > queries to the databases and avoid duplicating the same data in multiple 
> > places.
> 
> Fewer queries does not aklways make faster code, nor does it lead to 
> inherently race-free code.
> 
> > OpenStack is a big, distributed system of multiple databases that
> > sometimes rely on each other and cross-reference their records. It's
> > not uncommon to have some long-running operation started, that uses
> > some data, and then, in the middle of its execution, have that data deleted.
> > With soft delete, that's not a problem -- the operation can continue
> > safely and proceed as scheduled, with the data it was started with in
> > the first place -- it still has access to the deleted records as if
> > nothing happened.
> 
> I believe a better solution would be to use Boris' solution and implement 
> safeguards around the delete operation. For instance, not
> being able to delete an instance that has tasks still running against it. 
> Either that, or implement true task abortion logic that can
> notify distributed components about the need to stop a running task because 
> either the user wants to delete a resource or simply
> cancel the operation they began.
> 
> >  You simply won't be able to schedule another operation like that with
> > the same data, because it has been soft-deleted and won't pass the
> > validation at the beginning (or even won't appear in the UI or CLI).
> > This solves a lot of race conditions, error handling, additional
> > checks to make sure the record still exists, etc.
> 
> Sorry, I disagree here. Components that rely on the soft-delete behavior to 
> get the resource data from the database should instead
> respond to a NotFound that gets raised by aborting their running task.
> 
> > Without soft delete, you need to write custom code every time to
> > handle the case of a record being deleted mid-operation, including all
> > the possible combinations of which record and when.
> 
> N

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Joshua Harlow]

Off topic but, I'd like to see a word doc written out with the history of the 
cloud, that'd be pretty sweet.

Especially if its something like google docs where u can watch the changes 
happen in realtime.

+2

From: Jay Pipes mailto:jaypi...@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date: Friday, March 14, 2014 at 7:55 AM
To: 
"openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>" 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft 
deletion (step by step)

On Fri, 2014-03-14 at 08:37 +0100, Radomir Dopieralski wrote:
Hello,
I also think that this thread is going in the wrong direction, but I
don't think the direction Boris wants is the correct one either. Frankly
I'm a little surprised that nobody mentioned another advantage that soft
delete gives us, the one that I think it was actually used for originally.
You see, soft delete is an optimization. It's there to make the system
work faster as a whole, have less code and be simpler to maintain and debug.
How does it do it, when, as clearly shown in the first post in this
thread, it makes the queries slower, requires additional indices in the
database and more logic in the queries?

I feel it isn't an optimization if:

* It slows down the code base
* Makes the code harder to read and understand
* Deliberately obscures the actions of removing and restoring resources
* Encourages the idea that everything in the system is "undoable", like
the cloud is a Word doc.

  The answer is, by doing more
with those queries, by making you write less code, execute fewer queries
to the databases and avoid duplicating the same data in multiple places.

Fewer queries does not aklways make faster code, nor does it lead to
inherently race-free code.

OpenStack is a big, distributed system of multiple databases that
sometimes rely on each other and cross-reference their records. It's not
uncommon to have some long-running operation started, that uses some
data, and then, in the middle of its execution, have that data deleted.
With soft delete, that's not a problem -- the operation can continue
safely and proceed as scheduled, with the data it was started with in
the first place -- it still has access to the deleted records as if
nothing happened.

I believe a better solution would be to use Boris' solution and
implement safeguards around the delete operation. For instance, not
being able to delete an instance that has tasks still running against
it. Either that, or implement true task abortion logic that can notify
distributed components about the need to stop a running task because
either the user wants to delete a resource or simply cancel the
operation they began.

  You simply won't be able to schedule another operation
like that with the same data, because it has been soft-deleted and won't
pass the validation at the beginning (or even won't appear in the UI or
CLI). This solves a lot of race conditions, error handling, additional
checks to make sure the record still exists, etc.

Sorry, I disagree here. Components that rely on the soft-delete behavior
to get the resource data from the database should instead respond to a
NotFound that gets raised by aborting their running task.

Without soft delete, you need to write custom code every time to handle
the case of a record being deleted mid-operation, including all the
possible combinations of which record and when.

Not custom code. Explicit code paths for explicit actions.

  Or you need to copy all
the relevant data in advance over to whatever is executing that
operation.

This is already happening.

This cannot be abstracted away entirely (although tools like
TaskFlow help), as this is specific to the case you are handling. And
it's not easy to find all the places where you can have a race condition
like that -- especially when you are modifying existing code that has
been relying on soft delete before. You can have bugs undetected for
years, that only appear in production, on very large deployments, and
are impossible to reproduce reliably.
There are more similar cases like that, including cascading deletes and
more advanced stuff, but I think this single case already shows that
the advantages of soft delete out-weight its disadvantages.

I respectfully disagree :) I think the benefits of explicit code paths
and increased performance of the database outweigh the costs of changing
existing code.

Best,
-jay

On 13/03/14 19:52, Boris Pavlovic wrote:
> Hi all,
>
>
> I would like to fix direction of this thread. Cause it is going in wrong
> direction.
>
> To assume:
> 1) Yes restoring already deleted recourses could be useful.
> 2) Current approach with soft deletion is broken by design and we shoul

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Mike Wilson]

+1 to what Jay says here. This hidden behavior moistly just causes problems
and allows hacking hidden ways to restore things.

-Mike


On Fri, Mar 14, 2014 at 9:55 AM, Jay Pipes  wrote:

> On Fri, 2014-03-14 at 08:37 +0100, Radomir Dopieralski wrote:
> > Hello,
> >
> > I also think that this thread is going in the wrong direction, but I
> > don't think the direction Boris wants is the correct one either. Frankly
> > I'm a little surprised that nobody mentioned another advantage that soft
> > delete gives us, the one that I think it was actually used for
> originally.
> >
> > You see, soft delete is an optimization. It's there to make the system
> > work faster as a whole, have less code and be simpler to maintain and
> debug.
> >
> > How does it do it, when, as clearly shown in the first post in this
> > thread, it makes the queries slower, requires additional indices in the
> > database and more logic in the queries?
>
> I feel it isn't an optimization if:
>
> * It slows down the code base
> * Makes the code harder to read and understand
> * Deliberately obscures the actions of removing and restoring resources
> * Encourages the idea that everything in the system is "undoable", like
> the cloud is a Word doc.
>
> >  The answer is, by doing more
> > with those queries, by making you write less code, execute fewer queries
> > to the databases and avoid duplicating the same data in multiple places.
>
> Fewer queries does not aklways make faster code, nor does it lead to
> inherently race-free code.
>
> > OpenStack is a big, distributed system of multiple databases that
> > sometimes rely on each other and cross-reference their records. It's not
> > uncommon to have some long-running operation started, that uses some
> > data, and then, in the middle of its execution, have that data deleted.
> > With soft delete, that's not a problem -- the operation can continue
> > safely and proceed as scheduled, with the data it was started with in
> > the first place -- it still has access to the deleted records as if
> > nothing happened.
>
> I believe a better solution would be to use Boris' solution and
> implement safeguards around the delete operation. For instance, not
> being able to delete an instance that has tasks still running against
> it. Either that, or implement true task abortion logic that can notify
> distributed components about the need to stop a running task because
> either the user wants to delete a resource or simply cancel the
> operation they began.
>
> >  You simply won't be able to schedule another operation
> > like that with the same data, because it has been soft-deleted and won't
> > pass the validation at the beginning (or even won't appear in the UI or
> > CLI). This solves a lot of race conditions, error handling, additional
> > checks to make sure the record still exists, etc.
>
> Sorry, I disagree here. Components that rely on the soft-delete behavior
> to get the resource data from the database should instead respond to a
> NotFound that gets raised by aborting their running task.
>
> > Without soft delete, you need to write custom code every time to handle
> > the case of a record being deleted mid-operation, including all the
> > possible combinations of which record and when.
>
> Not custom code. Explicit code paths for explicit actions.
>
> >  Or you need to copy all
> > the relevant data in advance over to whatever is executing that
> > operation.
>
> This is already happening.
>
> > This cannot be abstracted away entirely (although tools like
> > TaskFlow help), as this is specific to the case you are handling. And
> > it's not easy to find all the places where you can have a race condition
> > like that -- especially when you are modifying existing code that has
> > been relying on soft delete before. You can have bugs undetected for
> > years, that only appear in production, on very large deployments, and
> > are impossible to reproduce reliably.
> >
> > There are more similar cases like that, including cascading deletes and
> > more advanced stuff, but I think this single case already shows that
> > the advantages of soft delete out-weight its disadvantages.
>
> I respectfully disagree :) I think the benefits of explicit code paths
> and increased performance of the database outweigh the costs of changing
> existing code.
>
> Best,
> -jay
>
> > On 13/03/14 19:52, Boris Pavlovic wrote:
> > > Hi all,
> > >
> > >
> > > I would like to fix direction of this thread. Cause it is going in
> wrong
> > > direction.
> > >
> > > To assume:
> > > 1) Yes restoring already deleted recourses could be useful.
> > > 2) Current approach with soft deletion is broken by design and we
> should
> > > get rid of them.
> > >
> > > More about why I think that it is broken:
> > > 1) When you are restoring some resource you should restore N records
> > > from N tables (e.g. VM)
> > > 2) Restoring sometimes means not only restoring DB records.
> > > 3) Not all resources should be restorable (e.g. why I

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Jay Pipes]

On Fri, 2014-03-14 at 08:37 +0100, Radomir Dopieralski wrote:
> Hello,
> 
> I also think that this thread is going in the wrong direction, but I
> don't think the direction Boris wants is the correct one either. Frankly
> I'm a little surprised that nobody mentioned another advantage that soft
> delete gives us, the one that I think it was actually used for originally.
> 
> You see, soft delete is an optimization. It's there to make the system
> work faster as a whole, have less code and be simpler to maintain and debug.
> 
> How does it do it, when, as clearly shown in the first post in this
> thread, it makes the queries slower, requires additional indices in the
> database and more logic in the queries?

I feel it isn't an optimization if:

* It slows down the code base
* Makes the code harder to read and understand
* Deliberately obscures the actions of removing and restoring resources
* Encourages the idea that everything in the system is "undoable", like
the cloud is a Word doc.

>  The answer is, by doing more
> with those queries, by making you write less code, execute fewer queries
> to the databases and avoid duplicating the same data in multiple places.

Fewer queries does not aklways make faster code, nor does it lead to
inherently race-free code.

> OpenStack is a big, distributed system of multiple databases that
> sometimes rely on each other and cross-reference their records. It's not
> uncommon to have some long-running operation started, that uses some
> data, and then, in the middle of its execution, have that data deleted.
> With soft delete, that's not a problem -- the operation can continue
> safely and proceed as scheduled, with the data it was started with in
> the first place -- it still has access to the deleted records as if
> nothing happened.

I believe a better solution would be to use Boris' solution and
implement safeguards around the delete operation. For instance, not
being able to delete an instance that has tasks still running against
it. Either that, or implement true task abortion logic that can notify
distributed components about the need to stop a running task because
either the user wants to delete a resource or simply cancel the
operation they began.

>  You simply won't be able to schedule another operation
> like that with the same data, because it has been soft-deleted and won't
> pass the validation at the beginning (or even won't appear in the UI or
> CLI). This solves a lot of race conditions, error handling, additional
> checks to make sure the record still exists, etc.

Sorry, I disagree here. Components that rely on the soft-delete behavior
to get the resource data from the database should instead respond to a
NotFound that gets raised by aborting their running task.

> Without soft delete, you need to write custom code every time to handle
> the case of a record being deleted mid-operation, including all the
> possible combinations of which record and when.

Not custom code. Explicit code paths for explicit actions.

>  Or you need to copy all
> the relevant data in advance over to whatever is executing that
> operation. 

This is already happening.

> This cannot be abstracted away entirely (although tools like
> TaskFlow help), as this is specific to the case you are handling. And
> it's not easy to find all the places where you can have a race condition
> like that -- especially when you are modifying existing code that has
> been relying on soft delete before. You can have bugs undetected for
> years, that only appear in production, on very large deployments, and
> are impossible to reproduce reliably.
> 
> There are more similar cases like that, including cascading deletes and
> more advanced stuff, but I think this single case already shows that
> the advantages of soft delete out-weight its disadvantages.

I respectfully disagree :) I think the benefits of explicit code paths
and increased performance of the database outweigh the costs of changing
existing code.

Best,
-jay

> On 13/03/14 19:52, Boris Pavlovic wrote:
> > Hi all, 
> > 
> > 
> > I would like to fix direction of this thread. Cause it is going in wrong
> > direction. 
> > 
> > To assume:
> > 1) Yes restoring already deleted recourses could be useful. 
> > 2) Current approach with soft deletion is broken by design and we should
> > get rid of them. 
> > 
> > More about why I think that it is broken: 
> > 1) When you are restoring some resource you should restore N records
> > from N tables (e.g. VM)
> > 2) Restoring sometimes means not only restoring DB records. 
> > 3) Not all resources should be restorable (e.g. why I need to restore
> > fixed_ip? or key-pairs?)
> > 
> > 
> > So what we should think about is:
> > 1) How to implement restoring functionally in common way (e.g. framework
> > that will be in oslo) 
> > 2) Split of work of getting rid of soft deletion in steps (that I
> > already mention):
> > a) remove soft deletion from places where we are not using it
> > b) replace int

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Radomir Dopieralski]

On 14/03/14 11:08, Alexei Kornienko wrote:
> On 03/14/2014 09:37 AM, Radomir Dopieralski wrote:

[snip]

>> OpenStack is a big, distributed system of multiple databases that
>> sometimes rely on each other and cross-reference their records. It's not
>> uncommon to have some long-running operation started, that uses some
>> data, and then, in the middle of its execution, have that data deleted.
>> With soft delete, that's not a problem -- the operation can continue
>> safely and proceed as scheduled, with the data it was started with in
>> the first place -- it still has access to the deleted records as if
>> nothing happened. You simply won't be able to schedule another operation
>> like that with the same data, because it has been soft-deleted and won't
>> pass the validation at the beginning (or even won't appear in the UI or
>> CLI). This solves a lot of race conditions, error handling, additional
>> checks to make sure the record still exists, etc.
>
> 1) Operation in SQL are working in transactions so deleted records will
> be visible for other clients until transaction commit.
>
> 2) If someone inside the same transaction will try to use record that is
> already deleted it's definitely an error in our code and should be fixed.
> I don't think that such use case can be used as an argument to keep soft
> deleted records.

Yes, that's why it works just fine when you have a single database in
one place. You can have locks, transactions, cascading operations and
all this stuff, and you have a guarantee that you are always in a
consistent state, unless there is a horrible bug.

OpenStack, however, is not a single database. There is no system-wide
solution for locks, transactions or rollbacks. Every time you reference
anything across databases, you are going to run into this problem.

-- 
Radomir Dopieralski

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Alexei Kornienko]

On 03/14/2014 09:37 AM, Radomir Dopieralski wrote:

Hello,

I also think that this thread is going in the wrong direction, but I
don't think the direction Boris wants is the correct one either. Frankly
I'm a little surprised that nobody mentioned another advantage that soft
delete gives us, the one that I think it was actually used for originally.

You see, soft delete is an optimization. It's there to make the system
work faster as a whole, have less code and be simpler to maintain and debug.

How does it do it, when, as clearly shown in the first post in this
thread, it makes the queries slower, requires additional indices in the
database and more logic in the queries? The answer is, by doing more
with those queries, by making you write less code, execute fewer queries
to the databases and avoid duplicating the same data in multiple places.

OpenStack is a big, distributed system of multiple databases that
sometimes rely on each other and cross-reference their records. It's not
uncommon to have some long-running operation started, that uses some
data, and then, in the middle of its execution, have that data deleted.
With soft delete, that's not a problem -- the operation can continue
safely and proceed as scheduled, with the data it was started with in
the first place -- it still has access to the deleted records as if
nothing happened. You simply won't be able to schedule another operation
like that with the same data, because it has been soft-deleted and won't
pass the validation at the beginning (or even won't appear in the UI or
CLI). This solves a lot of race conditions, error handling, additional
checks to make sure the record still exists, etc.
1) Operation in SQL are working in transactions so deleted records will 
be visible for other clients until transaction commit.
2) If someone inside the same transaction will try to use record that is 
already deleted it's definitely an error in our code and should be fixed.
I don't think that such use case can be used as an argument to keep soft 
deleted records.


Without soft delete, you need to write custom code every time to handle
the case of a record being deleted mid-operation, including all the
possible combinations of which record and when. Or you need to copy all
the relevant data in advance over to whatever is executing that
operation. This cannot be abstracted away entirely (although tools like
TaskFlow help), as this is specific to the case you are handling. And
it's not easy to find all the places where you can have a race condition
like that -- especially when you are modifying existing code that has
been relying on soft delete before. You can have bugs undetected for
years, that only appear in production, on very large deployments, and
are impossible to reproduce reliably.

There are more similar cases like that, including cascading deletes and
more advanced stuff, but I think this single case already shows that
the advantages of soft delete out-weight its disadvantages.

On 13/03/14 19:52, Boris Pavlovic wrote:

Hi all,


I would like to fix direction of this thread. Cause it is going in wrong
direction.

To assume:
1) Yes restoring already deleted recourses could be useful.
2) Current approach with soft deletion is broken by design and we should
get rid of them.

More about why I think that it is broken:
1) When you are restoring some resource you should restore N records
from N tables (e.g. VM)
2) Restoring sometimes means not only restoring DB records.
3) Not all resources should be restorable (e.g. why I need to restore
fixed_ip? or key-pairs?)


So what we should think about is:
1) How to implement restoring functionally in common way (e.g. framework
that will be in oslo)
2) Split of work of getting rid of soft deletion in steps (that I
already mention):
a) remove soft deletion from places where we are not using it
b) replace internal code where we are using soft deletion to that framework
c) replace API stuff using ceilometer (for logs) or this framework (for
restorable stuff)


To put in a nutshell: Restoring Delete resources / Delayed Deletion !=
Soft deletion.


Best regards,
Boris Pavlovic



On Thu, Mar 13, 2014 at 9:21 PM, Mike Wilson mailto:geekinu...@gmail.com>> wrote:

 For some guests we use the LVM imagebackend and there are times when
 the guest is deleted on accident. Humans, being what they are, don't
 back up their files and don't take care of important data, so it is
 not uncommon to use lvrestore and "undelete" an instance so that
 people can get their data. Of course, this is not always possible if
 the data has been subsequently overwritten. But it is common enough
 that I imagine most of our operators are familiar with how to do it.
 So I guess my saying that we do it on a regular basis is not quite
 accurate. Probably would be better to say that it is not uncommon to
 do this, but definitely not a daily task or something of that ilk.

 I have personally "undeleted" an 

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Radomir Dopieralski]

Hello,

I also think that this thread is going in the wrong direction, but I
don't think the direction Boris wants is the correct one either. Frankly
I'm a little surprised that nobody mentioned another advantage that soft
delete gives us, the one that I think it was actually used for originally.

You see, soft delete is an optimization. It's there to make the system
work faster as a whole, have less code and be simpler to maintain and debug.

How does it do it, when, as clearly shown in the first post in this
thread, it makes the queries slower, requires additional indices in the
database and more logic in the queries? The answer is, by doing more
with those queries, by making you write less code, execute fewer queries
to the databases and avoid duplicating the same data in multiple places.

OpenStack is a big, distributed system of multiple databases that
sometimes rely on each other and cross-reference their records. It's not
uncommon to have some long-running operation started, that uses some
data, and then, in the middle of its execution, have that data deleted.
With soft delete, that's not a problem -- the operation can continue
safely and proceed as scheduled, with the data it was started with in
the first place -- it still has access to the deleted records as if
nothing happened. You simply won't be able to schedule another operation
like that with the same data, because it has been soft-deleted and won't
pass the validation at the beginning (or even won't appear in the UI or
CLI). This solves a lot of race conditions, error handling, additional
checks to make sure the record still exists, etc.

Without soft delete, you need to write custom code every time to handle
the case of a record being deleted mid-operation, including all the
possible combinations of which record and when. Or you need to copy all
the relevant data in advance over to whatever is executing that
operation. This cannot be abstracted away entirely (although tools like
TaskFlow help), as this is specific to the case you are handling. And
it's not easy to find all the places where you can have a race condition
like that -- especially when you are modifying existing code that has
been relying on soft delete before. You can have bugs undetected for
years, that only appear in production, on very large deployments, and
are impossible to reproduce reliably.

There are more similar cases like that, including cascading deletes and
more advanced stuff, but I think this single case already shows that
the advantages of soft delete out-weight its disadvantages.

On 13/03/14 19:52, Boris Pavlovic wrote:
> Hi all, 
> 
> 
> I would like to fix direction of this thread. Cause it is going in wrong
> direction. 
> 
> To assume:
> 1) Yes restoring already deleted recourses could be useful. 
> 2) Current approach with soft deletion is broken by design and we should
> get rid of them. 
> 
> More about why I think that it is broken: 
> 1) When you are restoring some resource you should restore N records
> from N tables (e.g. VM)
> 2) Restoring sometimes means not only restoring DB records. 
> 3) Not all resources should be restorable (e.g. why I need to restore
> fixed_ip? or key-pairs?)
> 
> 
> So what we should think about is:
> 1) How to implement restoring functionally in common way (e.g. framework
> that will be in oslo) 
> 2) Split of work of getting rid of soft deletion in steps (that I
> already mention):
> a) remove soft deletion from places where we are not using it
> b) replace internal code where we are using soft deletion to that framework 
> c) replace API stuff using ceilometer (for logs) or this framework (for
> restorable stuff)
> 
> 
> To put in a nutshell: Restoring Delete resources / Delayed Deletion !=
> Soft deletion. 
> 
> 
> Best regards,
> Boris Pavlovic 
> 
> 
> 
> On Thu, Mar 13, 2014 at 9:21 PM, Mike Wilson  > wrote:
> 
> For some guests we use the LVM imagebackend and there are times when
> the guest is deleted on accident. Humans, being what they are, don't
> back up their files and don't take care of important data, so it is
> not uncommon to use lvrestore and "undelete" an instance so that
> people can get their data. Of course, this is not always possible if
> the data has been subsequently overwritten. But it is common enough
> that I imagine most of our operators are familiar with how to do it.
> So I guess my saying that we do it on a regular basis is not quite
> accurate. Probably would be better to say that it is not uncommon to
> do this, but definitely not a daily task or something of that ilk.
> 
> I have personally "undeleted" an instance a few times after
> accidental deletion also. I can't remember the specifics, but I do
> remember doing it :-).
> 
> -Mike
> 
> 
> On Tue, Mar 11, 2014 at 12:46 PM, Johannes Erdfelt
> mailto:johan...@erdfelt.com>> wrote:
> 
> On Tue, Mar 11, 2014, Mike Wilson  

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Mike Wilson]

The restore use case is for sure inconsistently implemented and used. I
think I agree with Boris that we treat it as separate and just move on with
cleaning up soft delete. I imagine most deployments don't like having most
of the rows in their table be useless and make db access slow? That being
said, I am a little sad my hacky restore method will need to be reworked
:-).

-Mike


On Thu, Mar 13, 2014 at 1:30 PM, Clint Byrum  wrote:

> Excerpts from Tim Bell's message of 2014-03-12 11:02:25 -0700:
> >
> > >
> > > If you want to archive images per-say, on deletion just export it to a
> 'backup tape' (for example) and store enough of the metadata
> > > on that 'tape' to re-insert it if this is really desired and then
> delete it from the database (or do the export... asynchronously). The
> > > same could be said with VMs, although likely not all resources, aka
> networks/.../ make sense to do this.
> > >
> > > So instead of deleted = 1, wait for cleaner, just save the resource (if
> > > possible) + enough metadata on some other system ('backup tape',
> alternate storage location, hdfs, ceph...) and leave it there unless
> > > it's really needed. Making the database more complex (and all
> associated code) to achieve this same goal seems like a hack that just
> > > needs to be addressed with a better way to do archiving.
> > >
> > > In a cloudy world of course people would be able to recreate
> everything they need on-demand so who needs undelete anyway ;-)
> > >
> >
> > I have no problem if there was an existing process integrated into all
> of the OpenStack components which would produce me an archive trail with
> meta data and a command to recover the object from that data.
> >
> > Currently, my understanding is that there is no such function and thus
> the proposal to remove the deleted column is premature.
> >
>
> That seems like an unreasonable request of low level tools like Nova. End
> user applications and infrastructure management should be responsible
> for these things and will do a much better job of it, as you can work
> your own business needs for reliability and recovery speed into an
> application aware solution. If Nova does it, your cloud just has to
> provide everybody with the same un-delete, which is probably overkill
> for _many_ applications.
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Clint Byrum]

Excerpts from Tim Bell's message of 2014-03-12 11:02:25 -0700:
> 
> > 
> > If you want to archive images per-say, on deletion just export it to a 
> > 'backup tape' (for example) and store enough of the metadata
> > on that 'tape' to re-insert it if this is really desired and then delete it 
> > from the database (or do the export... asynchronously). The
> > same could be said with VMs, although likely not all resources, aka 
> > networks/.../ make sense to do this.
> > 
> > So instead of deleted = 1, wait for cleaner, just save the resource (if
> > possible) + enough metadata on some other system ('backup tape', alternate 
> > storage location, hdfs, ceph...) and leave it there unless
> > it's really needed. Making the database more complex (and all associated 
> > code) to achieve this same goal seems like a hack that just
> > needs to be addressed with a better way to do archiving.
> > 
> > In a cloudy world of course people would be able to recreate everything 
> > they need on-demand so who needs undelete anyway ;-)
> > 
> 
> I have no problem if there was an existing process integrated into all of the 
> OpenStack components which would produce me an archive trail with meta data 
> and a command to recover the object from that data.
> 
> Currently, my understanding is that there is no such function and thus the 
> proposal to remove the deleted column is premature.
> 

That seems like an unreasonable request of low level tools like Nova. End
user applications and infrastructure management should be responsible
for these things and will do a much better job of it, as you can work
your own business needs for reliability and recovery speed into an
application aware solution. If Nova does it, your cloud just has to
provide everybody with the same un-delete, which is probably overkill
for _many_ applications.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Clint Byrum]

Excerpts from Jay Pipes's message of 2014-03-12 10:58:36 -0700:
> On Wed, 2014-03-12 at 17:35 +, Tim Bell wrote:
> > And if the same mistake is done for a cinder volume or a trove database ?
> 
> Snapshots and backups?
> 

and bears, oh my!

+1, whether it is large data on a volume or saved state in the RAM of
a compute node, it isn't safe unless it is duplicated.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Boris Pavlovic]

Hi all,


I would like to fix direction of this thread. Cause it is going in wrong
direction.

To assume:
1) Yes restoring already deleted recourses could be useful.
2) Current approach with soft deletion is broken by design and we should
get rid of them.

More about why I think that it is broken:
1) When you are restoring some resource you should restore N records from N
tables (e.g. VM)
2) Restoring sometimes means not only restoring DB records.
3) Not all resources should be restorable (e.g. why I need to restore
fixed_ip? or key-pairs?)


So what we should think about is:
1) How to implement restoring functionally in common way (e.g. framework
that will be in oslo)
2) Split of work of getting rid of soft deletion in steps (that I already
mention):
a) remove soft deletion from places where we are not using it
b) replace internal code where we are using soft deletion to that framework
c) replace API stuff using ceilometer (for logs) or this framework (for
restorable stuff)


To put in a nutshell: Restoring Delete resources / Delayed Deletion != Soft
deletion.


Best regards,
Boris Pavlovic



On Thu, Mar 13, 2014 at 9:21 PM, Mike Wilson  wrote:

> For some guests we use the LVM imagebackend and there are times when the
> guest is deleted on accident. Humans, being what they are, don't back up
> their files and don't take care of important data, so it is not uncommon to
> use lvrestore and "undelete" an instance so that people can get their data.
> Of course, this is not always possible if the data has been subsequently
> overwritten. But it is common enough that I imagine most of our operators
> are familiar with how to do it. So I guess my saying that we do it on a
> regular basis is not quite accurate. Probably would be better to say that
> it is not uncommon to do this, but definitely not a daily task or something
> of that ilk.
>
> I have personally "undeleted" an instance a few times after accidental
> deletion also. I can't remember the specifics, but I do remember doing it
> :-).
>
> -Mike
>
>
> On Tue, Mar 11, 2014 at 12:46 PM, Johannes Erdfelt 
> wrote:
>
>> On Tue, Mar 11, 2014, Mike Wilson  wrote:
>> > Undeleting things is an important use case in my opinion. We do this in
>> our
>> > environment on a regular basis. In that light I'm not sure that it
>> would be
>> > appropriate just to log the deletion and git rid of the row. I would
>> like
>> > to see it go to an archival table where it is easily restored.
>>
>> I'm curious, what are you undeleting and why?
>>
>> JE
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Mike Wilson]

For some guests we use the LVM imagebackend and there are times when the
guest is deleted on accident. Humans, being what they are, don't back up
their files and don't take care of important data, so it is not uncommon to
use lvrestore and "undelete" an instance so that people can get their data.
Of course, this is not always possible if the data has been subsequently
overwritten. But it is common enough that I imagine most of our operators
are familiar with how to do it. So I guess my saying that we do it on a
regular basis is not quite accurate. Probably would be better to say that
it is not uncommon to do this, but definitely not a daily task or something
of that ilk.

I have personally "undeleted" an instance a few times after accidental
deletion also. I can't remember the specifics, but I do remember doing it
:-).

-Mike


On Tue, Mar 11, 2014 at 12:46 PM, Johannes Erdfelt wrote:

> On Tue, Mar 11, 2014, Mike Wilson  wrote:
> > Undeleting things is an important use case in my opinion. We do this in
> our
> > environment on a regular basis. In that light I'm not sure that it would
> be
> > appropriate just to log the deletion and git rid of the row. I would like
> > to see it go to an archival table where it is easily restored.
>
> I'm curious, what are you undeleting and why?
>
> JE
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Duncan Thomas]

On 12 March 2014 17:35, Tim Bell  wrote:

> And if the same mistake is done for a cinder volume or a trove database ?

Deferred deletion for cinder has been proposed, and there have been
few objections to it... nobody has put forward code yet, but anybody
is welcome to do so.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Tim Bell]

> 
> If you want to archive images per-say, on deletion just export it to a 
> 'backup tape' (for example) and store enough of the metadata
> on that 'tape' to re-insert it if this is really desired and then delete it 
> from the database (or do the export... asynchronously). The
> same could be said with VMs, although likely not all resources, aka 
> networks/.../ make sense to do this.
> 
> So instead of deleted = 1, wait for cleaner, just save the resource (if
> possible) + enough metadata on some other system ('backup tape', alternate 
> storage location, hdfs, ceph...) and leave it there unless
> it's really needed. Making the database more complex (and all associated 
> code) to achieve this same goal seems like a hack that just
> needs to be addressed with a better way to do archiving.
> 
> In a cloudy world of course people would be able to recreate everything they 
> need on-demand so who needs undelete anyway ;-)
> 

I have no problem if there was an existing process integrated into all of the 
OpenStack components which would produce me an archive trail with meta data and 
a command to recover the object from that data.

Currently, my understanding is that there is no such function and thus the 
proposal to remove the deleted column is premature.

Tim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Jay Pipes]

On Wed, 2014-03-12 at 17:35 +, Tim Bell wrote:
> And if the same mistake is done for a cinder volume or a trove database ?

Snapshots and backups?

Best,
-jay

> 
> > -Original Message-
> > From: Joshua Harlow [mailto:harlo...@yahoo-inc.com]
> > Sent: 12 March 2014 17:02
> > To: OpenStack Development Mailing List (not for usage questions)
> > Cc: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid of 
> > soft deletion (step by step)
> > 
> > Understandable,
> > 
> > Humans will be humans after all.
> > 
> > To me if openstsck is a cloud platform then coming along with it should be 
> > best practices that come with the usage of a cloud
> > platform (treat your instances as ephemeral, use configuration management, 
> > save your stuff in source control...). I have been
> > preaching similar stuff at y! and getting people into the right mindset 
> > around "the cloud" is IMHO more important than making
> > openstack fit peoples non-cloudy mindset.
> > 
> > Because once u teach a person to use the cloud right u don't need to have 
> > openstack compensate for them using it incorrectly.
> > 
> > Sent from my really tiny device...
> > 
> > > On Mar 12, 2014, at 4:45 AM, "CARVER, PAUL"  wrote:
> > >
> > > I have personally witnessed someone (honestly, not me) select "Terminate 
> > > Instance" when they meant "Reboot Instance" and that
> > mistake is way too easy. I'm not sure if it was a brain mistake or mere 
> > slip of the mouse, but it's enough to make people really
> > nervous in a production environment. If there's one thing you can count on 
> > about human beings, it's that they'll make mistakes
> > sooner or later. Any system that assumes infallible human beings as a 
> > design criteria is making an invalid assumption.
> > >
> > > --
> > > Paul Carver
> > > VO: 732-545-7377
> > > Cell: 908-803-1656
> > > E: pcar...@att.com
> > > Q Instant Message
> > >
> > >
> > > -Original Message-
> > > From: Tim Bell [mailto:tim.b...@cern.ch]
> > > Sent: Tuesday, March 11, 2014 15:43
> > > To: OpenStack Development Mailing List (not for usage questions)
> > > Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid
> > > of soft deletion (step by step)
> > >
> > >
> > > Typical cases are user error where someone accidentally deletes an item 
> > > from a tenant. The image guys have a good structure
> > where images become unavailable and are recoverable for a certain period of 
> > time. A regular periodic task cleans up deleted items
> > after a configurable number of seconds to avoid constant database growth.
> > >
> > > My preference would be to follow this model universally (an archive table 
> > > is a nice way to do it without disturbing production).
> > >
> > > Tim
> > >
> > >
> > >>> On Tue, Mar 11, 2014, Mike Wilson  wrote:
> > >>> Undeleting things is an important use case in my opinion. We do this
> > >>> in our environment on a regular basis. In that light I'm not sure
> > >>> that it would be appropriate just to log the deletion and git rid of
> > >>> the row. I would like to see it go to an archival table where it is 
> > >>> easily restored.
> > >>
> > >> I'm curious, what are you undeleting and why?
> > >>
> > >> JE
> > >>
> > >>
> > >> ___
> > >> OpenStack-dev mailing list
> > >> OpenStack-dev@lists.openstack.org
> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > > ___
> > > OpenStack-dev mailing list
> > > OpenStack-dev@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > > ___
> > > OpenStack-dev mailing list
> > > OpenStack-dev@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Tim Bell]

And if the same mistake is done for a cinder volume or a trove database ?

Tim

> -Original Message-
> From: Joshua Harlow [mailto:harlo...@yahoo-inc.com]
> Sent: 12 March 2014 17:02
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft 
> deletion (step by step)
> 
> Understandable,
> 
> Humans will be humans after all.
> 
> To me if openstsck is a cloud platform then coming along with it should be 
> best practices that come with the usage of a cloud
> platform (treat your instances as ephemeral, use configuration management, 
> save your stuff in source control...). I have been
> preaching similar stuff at y! and getting people into the right mindset 
> around "the cloud" is IMHO more important than making
> openstack fit peoples non-cloudy mindset.
> 
> Because once u teach a person to use the cloud right u don't need to have 
> openstack compensate for them using it incorrectly.
> 
> Sent from my really tiny device...
> 
> > On Mar 12, 2014, at 4:45 AM, "CARVER, PAUL"  wrote:
> >
> > I have personally witnessed someone (honestly, not me) select "Terminate 
> > Instance" when they meant "Reboot Instance" and that
> mistake is way too easy. I'm not sure if it was a brain mistake or mere slip 
> of the mouse, but it's enough to make people really
> nervous in a production environment. If there's one thing you can count on 
> about human beings, it's that they'll make mistakes
> sooner or later. Any system that assumes infallible human beings as a design 
> criteria is making an invalid assumption.
> >
> > --
> > Paul Carver
> > VO: 732-545-7377
> > Cell: 908-803-1656
> > E: pcar...@att.com
> > Q Instant Message
> >
> >
> > -----Original Message-
> > From: Tim Bell [mailto:tim.b...@cern.ch]
> > Sent: Tuesday, March 11, 2014 15:43
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid
> > of soft deletion (step by step)
> >
> >
> > Typical cases are user error where someone accidentally deletes an item 
> > from a tenant. The image guys have a good structure
> where images become unavailable and are recoverable for a certain period of 
> time. A regular periodic task cleans up deleted items
> after a configurable number of seconds to avoid constant database growth.
> >
> > My preference would be to follow this model universally (an archive table 
> > is a nice way to do it without disturbing production).
> >
> > Tim
> >
> >
> >>> On Tue, Mar 11, 2014, Mike Wilson  wrote:
> >>> Undeleting things is an important use case in my opinion. We do this
> >>> in our environment on a regular basis. In that light I'm not sure
> >>> that it would be appropriate just to log the deletion and git rid of
> >>> the row. I would like to see it go to an archival table where it is 
> >>> easily restored.
> >>
> >> I'm curious, what are you undeleting and why?
> >>
> >> JE
> >>
> >>
> >> ___
> >> OpenStack-dev mailing list
> >> OpenStack-dev@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Johannes Erdfelt]

On Wed, Mar 12, 2014, CARVER, PAUL  wrote:
> I have personally witnessed someone (honestly, not me) select "Terminate
> Instance" when they meant "Reboot Instance" and that mistake is way too
> easy. I'm not sure if it was a brain mistake or mere slip of the mouse,
> but it's enough to make people really nervous in a production
> environment. If there's one thing you can count on about human beings,
> it's that they'll make mistakes sooner or later. Any system that
> assumes infallible human beings as a design criteria is making an
> invalid assumption.

I think there might be some confusion about what soft-delete we're
talking about.

Nova has two orthogonal "soft-delete" features:
1) Database rows are never deleted from the database. They are just
   marked as deleted via a column. This is unexposed to users and is an
   implementation detail in the current code.
2) Instance deletion can be deferred until a later time. This is called
   deferred-delete and soft-delete in the code. If the feature is
   enabled and the instance that has't been reclaimed, it can be
   restored with the 'nova restore' command.

This thread is about the database soft-delete and not the instance
soft-delete.

JE


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Jay Pipes]

On Wed, 2014-03-12 at 11:37 +, CARVER, PAUL wrote:
> I have personally witnessed someone (honestly, not me) select "Terminate 
> Instance" when they meant "Reboot Instance" and that mistake is way too easy. 
> I'm not sure if it was a brain mistake or mere slip of the mouse, but it's 
> enough to make people really nervous in a production environment. If there's 
> one thing you can count on about human beings, it's that they'll make 
> mistakes sooner or later. Any system that assumes infallible human beings as 
> a design criteria is making an invalid assumption.

That's why GUIs should have a dialog box that says "Are you sure you
want to terminate this server?".

There's prevention of common mistakes, and then there's going out of
your way to ensure that the cloud acts like a text editor with an
unlimiited undo buffer.

Best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Joshua Harlow]

Understandable,

Humans will be humans after all. 

To me if openstsck is a cloud platform then coming along with it should be best 
practices that come with the usage of a cloud platform (treat your instances as 
ephemeral, use configuration management, save your stuff in source control...). 
I have been preaching similar stuff at y! and getting people into the right 
mindset around "the cloud" is IMHO more important than making openstack fit 
peoples non-cloudy mindset.

Because once u teach a person to use the cloud right u don't need to have 
openstack compensate for them using it incorrectly.

Sent from my really tiny device...

> On Mar 12, 2014, at 4:45 AM, "CARVER, PAUL"  wrote:
> 
> I have personally witnessed someone (honestly, not me) select "Terminate 
> Instance" when they meant "Reboot Instance" and that mistake is way too easy. 
> I'm not sure if it was a brain mistake or mere slip of the mouse, but it's 
> enough to make people really nervous in a production environment. If there's 
> one thing you can count on about human beings, it's that they'll make 
> mistakes sooner or later. Any system that assumes infallible human beings as 
> a design criteria is making an invalid assumption.
> 
> -- 
> Paul Carver
> VO: 732-545-7377
> Cell: 908-803-1656
> E: pcar...@att.com
> Q Instant Message
> 
> 
> -Original Message-
> From: Tim Bell [mailto:tim.b...@cern.ch] 
> Sent: Tuesday, March 11, 2014 15:43
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft 
> deletion (step by step)
> 
> 
> Typical cases are user error where someone accidentally deletes an item from 
> a tenant. The image guys have a good structure where images become 
> unavailable and are recoverable for a certain period of time. A regular 
> periodic task cleans up deleted items after a configurable number of seconds 
> to avoid constant database growth.
> 
> My preference would be to follow this model universally (an archive table is 
> a nice way to do it without disturbing production).
> 
> Tim
> 
> 
>>> On Tue, Mar 11, 2014, Mike Wilson  wrote:
>>> Undeleting things is an important use case in my opinion. We do this
>>> in our environment on a regular basis. In that light I'm not sure that
>>> it would be appropriate just to log the deletion and git rid of the
>>> row. I would like to see it go to an archival table where it is easily 
>>> restored.
>> 
>> I'm curious, what are you undeleting and why?
>> 
>> JE
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[CARVER, PAUL]

I have personally witnessed someone (honestly, not me) select "Terminate 
Instance" when they meant "Reboot Instance" and that mistake is way too easy. 
I'm not sure if it was a brain mistake or mere slip of the mouse, but it's 
enough to make people really nervous in a production environment. If there's 
one thing you can count on about human beings, it's that they'll make mistakes 
sooner or later. Any system that assumes infallible human beings as a design 
criteria is making an invalid assumption.

-- 
Paul Carver
VO: 732-545-7377
Cell: 908-803-1656
E: pcar...@att.com
Q Instant Message


-Original Message-
From: Tim Bell [mailto:tim.b...@cern.ch] 
Sent: Tuesday, March 11, 2014 15:43
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft 
deletion (step by step)


Typical cases are user error where someone accidentally deletes an item from a 
tenant. The image guys have a good structure where images become unavailable 
and are recoverable for a certain period of time. A regular periodic task 
cleans up deleted items after a configurable number of seconds to avoid 
constant database growth.

My preference would be to follow this model universally (an archive table is a 
nice way to do it without disturbing production).

Tim


> On Tue, Mar 11, 2014, Mike Wilson  wrote:
> > Undeleting things is an important use case in my opinion. We do this
> > in our environment on a regular basis. In that light I'm not sure that
> > it would be appropriate just to log the deletion and git rid of the
> > row. I would like to see it go to an archival table where it is easily 
> > restored.
> 
> I'm curious, what are you undeleting and why?
> 
> JE
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Sean Dague]

On 03/11/2014 11:10 PM, Jay Pipes wrote:
> On Wed, 2014-03-12 at 01:47 +, Joshua Harlow wrote:
>> The question that I don't understand is why does this process have to be
>> involve the database to begin with?
>>
>> If you want to archive images per-say, on deletion just export it to a
>> 'backup tape' (for example) and store enough of the metadata on that
>> 'tape' to re-insert it if this is really desired and then delete it from
>> the database (or do the export... asynchronously). The same could be said
>> with VMs, although likely not all resources, aka networks/.../ make sense
>> to do this.
>>
>> So instead of deleted = 1, wait for cleaner, just save the resource (if
>> possible) + enough metadata on some other system ('backup tape', alternate
>> storage location, hdfs, ceph...) and leave it there unless it's really
>> needed. Making the database more complex (and all associated code) to
>> achieve this same goal seems like a hack that just needs to be addressed
>> with a better way to do archiving.
>>
>> In a cloudy world of course people would be able to recreate everything
>> they need on-demand so who needs undelete anyway ;-)
> 
> Good points.
> 
> Another way to ask the question: does Amazon provide an undelete
> functionality?

Man, if that was our threshold for doing things, we could delete a ton
of OpenStack code. :P

Honestly, I think it's important to realize that a very large OpenStack
deploy has found undelete *really useful*. Perhaps the current way they
are doing it is hacky based on something that wasn't intended, so we
should have a more explicit undelete method, however I don't think it's
immediately invalid because AWS doesn't do it.

Back to the original question, my, limited, understanding is soft delete
is there for eventual consistency. It means that everyone doesn't need a
current view of the database all the time, and that a bunch of
operations don't need to sit under transactions. Like the ability to do
a list_images while and image is being deleted, especially given that we
denormalize a lot of these things over multiple tables. That, however,
may be an out of date concept here. I haven't stayed up on everything in
that piece of the stack.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Jay Pipes]

On Wed, 2014-03-12 at 01:47 +, Joshua Harlow wrote:
> The question that I don't understand is why does this process have to be
> involve the database to begin with?
> 
> If you want to archive images per-say, on deletion just export it to a
> 'backup tape' (for example) and store enough of the metadata on that
> 'tape' to re-insert it if this is really desired and then delete it from
> the database (or do the export... asynchronously). The same could be said
> with VMs, although likely not all resources, aka networks/.../ make sense
> to do this.
> 
> So instead of deleted = 1, wait for cleaner, just save the resource (if
> possible) + enough metadata on some other system ('backup tape', alternate
> storage location, hdfs, ceph...) and leave it there unless it's really
> needed. Making the database more complex (and all associated code) to
> achieve this same goal seems like a hack that just needs to be addressed
> with a better way to do archiving.
> 
> In a cloudy world of course people would be able to recreate everything
> they need on-demand so who needs undelete anyway ;-)

Good points.

Another way to ask the question: does Amazon provide an undelete
functionality?

Best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Joshua Harlow]

The question that I don't understand is why does this process have to be
involve the database to begin with?

If you want to archive images per-say, on deletion just export it to a
'backup tape' (for example) and store enough of the metadata on that
'tape' to re-insert it if this is really desired and then delete it from
the database (or do the export... asynchronously). The same could be said
with VMs, although likely not all resources, aka networks/.../ make sense
to do this.

So instead of deleted = 1, wait for cleaner, just save the resource (if
possible) + enough metadata on some other system ('backup tape', alternate
storage location, hdfs, ceph...) and leave it there unless it's really
needed. Making the database more complex (and all associated code) to
achieve this same goal seems like a hack that just needs to be addressed
with a better way to do archiving.

In a cloudy world of course people would be able to recreate everything
they need on-demand so who needs undelete anyway ;-)

My 0.02 cents.

-Original Message-
From: Tim Bell 
Reply-To: "OpenStack Development Mailing List (not for usage questions)"

Date: Tuesday, March 11, 2014 at 11:43 AM
To: "OpenStack Development Mailing List (not for usage questions)"

Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid of
soft deletion (step by step)

>
>Typical cases are user error where someone accidentally deletes an item
>from a tenant. The image guys have a good structure where images become
>unavailable and are recoverable for a certain period of time. A regular
>periodic task cleans up deleted items after a configurable number of
>seconds to avoid constant database growth.
>
>My preference would be to follow this model universally (an archive table
>is a nice way to do it without disturbing production).
>
>Tim
>
>
>> On Tue, Mar 11, 2014, Mike Wilson  wrote:
>> > Undeleting things is an important use case in my opinion. We do this
>> > in our environment on a regular basis. In that light I'm not sure that
>> > it would be appropriate just to log the deletion and git rid of the
>> > row. I would like to see it go to an archival table where it is
>>easily restored.
>> 
>> I'm curious, what are you undeleting and why?
>> 
>> JE
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Tim Bell]

Can we therefore make that no removal of deleted column is permitted if there 
is no implementation of shadow tables ?

Tim

From: Joe Gordon [mailto:joe.gord...@gmail.com]
Sent: 11 March 2014 20:57
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft 
deletion (step by step)



On Tue, Mar 11, 2014 at 12:43 PM, Tim Bell 
mailto:tim.b...@cern.ch>> wrote:

Typical cases are user error where someone accidentally deletes an item from a 
tenant. The image guys have a good structure where images become unavailable 
and are recoverable for a certain period of time. A regular periodic task 
cleans up deleted items after a configurable number of seconds to avoid 
constant database growth.

My preference would be to follow this model universally (an archive table is a 
nice way to do it without disturbing production).

That was the goal of the shadow table, if it doesn't support that now then its 
a bug.


Tim


> On Tue, Mar 11, 2014, Mike Wilson 
> mailto:geekinu...@gmail.com>> wrote:
> > Undeleting things is an important use case in my opinion. We do this
> > in our environment on a regular basis. In that light I'm not sure that
> > it would be appropriate just to log the deletion and git rid of the
> > row. I would like to see it go to an archival table where it is easily 
> > restored.
>
> I'm curious, what are you undeleting and why?
>
> JE
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Joe Gordon]

On Tue, Mar 11, 2014 at 12:43 PM, Tim Bell  wrote:

>
> Typical cases are user error where someone accidentally deletes an item
> from a tenant. The image guys have a good structure where images become
> unavailable and are recoverable for a certain period of time. A regular
> periodic task cleans up deleted items after a configurable number of
> seconds to avoid constant database growth.
>
> My preference would be to follow this model universally (an archive table
> is a nice way to do it without disturbing production).
>

That was the goal of the shadow table, if it doesn't support that now then
its a bug.


>
> Tim
>
>
> > On Tue, Mar 11, 2014, Mike Wilson  wrote:
> > > Undeleting things is an important use case in my opinion. We do this
> > > in our environment on a regular basis. In that light I'm not sure that
> > > it would be appropriate just to log the deletion and git rid of the
> > > row. I would like to see it go to an archival table where it is easily
> restored.
> >
> > I'm curious, what are you undeleting and why?
> >
> > JE
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Tim Bell]

Typical cases are user error where someone accidentally deletes an item from a 
tenant. The image guys have a good structure where images become unavailable 
and are recoverable for a certain period of time. A regular periodic task 
cleans up deleted items after a configurable number of seconds to avoid 
constant database growth.

My preference would be to follow this model universally (an archive table is a 
nice way to do it without disturbing production).

Tim


> On Tue, Mar 11, 2014, Mike Wilson  wrote:
> > Undeleting things is an important use case in my opinion. We do this
> > in our environment on a regular basis. In that light I'm not sure that
> > it would be appropriate just to log the deletion and git rid of the
> > row. I would like to see it go to an archival table where it is easily 
> > restored.
> 
> I'm curious, what are you undeleting and why?
> 
> JE
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Johannes Erdfelt]

On Tue, Mar 11, 2014, Mike Wilson  wrote:
> Undeleting things is an important use case in my opinion. We do this in our
> environment on a regular basis. In that light I'm not sure that it would be
> appropriate just to log the deletion and git rid of the row. I would like
> to see it go to an archival table where it is easily restored.

I'm curious, what are you undeleting and why?

JE


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Joe Gordon]

On Tue, Mar 11, 2014 at 10:24 AM, Mike Wilson  wrote:

> Undeleting things is an important use case in my opinion. We do this in
> our environment on a regular basis. In that light I'm not sure that it
> would be appropriate just to log the deletion and git rid of the row. I
> would like to see it go to an archival table where it is easily restored.
>
>
Although we want to *support* hard deletion, we still want to support the
current behavior as well (Soft deletion, where the operator, can prune
deleted rows periodically).


> -Mike
>
>
> On Mon, Mar 10, 2014 at 3:44 PM, Joshua Harlow wrote:
>
>>  Sounds like a good idea to me.
>>
>>  I've never understood why we treat the DB as a LOG (keeping deleted ==
>> 0 records around) when we should just use a LOG (or similar system) to
>> begin with instead.
>>
>>  Does anyone use the feature of switching deleted == 1 back to deleted =
>> 0? Has this worked out for u?
>>
>>  Seems like some of the feedback on
>> https://etherpad.openstack.org/p/operators-feedback-mar14 also suggests
>> that this has been a operational pain-point for folks (Tool to delete
>> things properly suggestions and such...).
>>
>>   From: Boris Pavlovic 
>> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
>> 
>> Date: Monday, March 10, 2014 at 1:29 PM
>> To: OpenStack Development Mailing List ,
>> Victor Sergeyev 
>> Subject: [openstack-dev] [all][db][performance] Proposal: Get rid of
>> soft deletion (step by step)
>>
>>   Hi stackers,
>>
>>  (It's proposal for Juno.)
>>
>>  Intro:
>>
>>  Soft deletion means that records from DB are not actually deleted, they
>> are just marked as a "deleted". To mark record as a "deleted" we put in
>> special table's column "deleted" record's ID value.
>>
>>  Issue 1: Indexes & Queries
>> We have to add in every query "AND deleted == 0" to get non-deleted
>> records.
>> It produce performance issue, cause we should add it in any index one
>> "extra" column.
>> As well it produce extra complexity in db migrations and building
>> queries.
>>
>>  Issue 2: Unique constraints
>> Why we store ID in deleted and not True/False?
>>  The reason is that we would like to be able to create real DB unique
>> constraints and avoid race conditions on "insert" operation.
>>
>>  Sample: we Have table (id, name, password, deleted) we would like to
>> put in column "name" only unique value.
>>
>>  Approach without UC: if count(`select  where name = name`) == 0:
>> insert(...)
>> (race cause we are able to add new record between )
>>
>>  Approach with UC: try: insert(...) except Duplicate: ...
>>
>>  So to add UC we have to add them on (name, deleted). (to be able to
>> make insert/delete/insert with same name)
>>
>>  As well it produce performance issues, because we have to use Complex
>> unique constraints on 2  or more columns. + extra code & complexity in db
>> migrations.
>>
>>  Issue 3: Garbage collector
>>
>>  It is really hard to make garbage collector that will have good
>> performance and be enough common to work in any case for any project.
>> Without garbage collector DevOps have to cleanup records by hand, (risk
>> to break something). If they don't cleanup DB they will get very soon
>> performance issue.
>>
>>  To put in a nutshell most important issues:
>> 1) Extra complexity to each select query & extra column in each index
>> 2) Extra column in each Unique Constraint (worse performance)
>> 3) 2 Extra column in each table: (deleted, deleted_at)
>> 4) Common garbage collector is required
>>
>>
>>  To resolve all these issues we should just remove soft deletion.
>>
>>  One of approaches that I see is in step by step removing "deleted"
>> column from every table with probably code refactoring.  Actually we have 3
>> different cases:
>>
>>  1) We don't use soft deleted records:
>> 1.1) Do .delete() instead of .soft_delete()
>> 1.2) Change query to avoid adding extra "deleted == 0" to each query
>> 1.3) Drop "deleted" and "deleted_at" columns
>>
>>  2) We use soft deleted records for internal stuff "e.g. periodic tasks"
>> 2.1) Refactor code somehow: E.g. store all required data by periodic task
>> in some special table that has: (id

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Mike Wilson]

Undeleting things is an important use case in my opinion. We do this in our
environment on a regular basis. In that light I'm not sure that it would be
appropriate just to log the deletion and git rid of the row. I would like
to see it go to an archival table where it is easily restored.

-Mike


On Mon, Mar 10, 2014 at 3:44 PM, Joshua Harlow wrote:

>  Sounds like a good idea to me.
>
>  I've never understood why we treat the DB as a LOG (keeping deleted == 0
> records around) when we should just use a LOG (or similar system) to begin
> with instead.
>
>  Does anyone use the feature of switching deleted == 1 back to deleted =
> 0? Has this worked out for u?
>
>  Seems like some of the feedback on
> https://etherpad.openstack.org/p/operators-feedback-mar14 also suggests
> that this has been a operational pain-point for folks (Tool to delete
> things properly suggestions and such...).
>
>   From: Boris Pavlovic 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Monday, March 10, 2014 at 1:29 PM
> To: OpenStack Development Mailing List ,
> Victor Sergeyev 
> Subject: [openstack-dev] [all][db][performance] Proposal: Get rid of soft
> deletion (step by step)
>
>   Hi stackers,
>
>  (It's proposal for Juno.)
>
>  Intro:
>
>  Soft deletion means that records from DB are not actually deleted, they
> are just marked as a "deleted". To mark record as a "deleted" we put in
> special table's column "deleted" record's ID value.
>
>  Issue 1: Indexes & Queries
> We have to add in every query "AND deleted == 0" to get non-deleted
> records.
> It produce performance issue, cause we should add it in any index one
> "extra" column.
> As well it produce extra complexity in db migrations and building queries.
>
>  Issue 2: Unique constraints
> Why we store ID in deleted and not True/False?
>  The reason is that we would like to be able to create real DB unique
> constraints and avoid race conditions on "insert" operation.
>
>  Sample: we Have table (id, name, password, deleted) we would like to put
> in column "name" only unique value.
>
>  Approach without UC: if count(`select  where name = name`) == 0:
> insert(...)
> (race cause we are able to add new record between )
>
>  Approach with UC: try: insert(...) except Duplicate: ...
>
>  So to add UC we have to add them on (name, deleted). (to be able to make
> insert/delete/insert with same name)
>
>  As well it produce performance issues, because we have to use Complex
> unique constraints on 2  or more columns. + extra code & complexity in db
> migrations.
>
>  Issue 3: Garbage collector
>
>  It is really hard to make garbage collector that will have good
> performance and be enough common to work in any case for any project.
> Without garbage collector DevOps have to cleanup records by hand, (risk to
> break something). If they don't cleanup DB they will get very soon
> performance issue.
>
>  To put in a nutshell most important issues:
> 1) Extra complexity to each select query & extra column in each index
> 2) Extra column in each Unique Constraint (worse performance)
> 3) 2 Extra column in each table: (deleted, deleted_at)
> 4) Common garbage collector is required
>
>
>  To resolve all these issues we should just remove soft deletion.
>
>  One of approaches that I see is in step by step removing "deleted"
> column from every table with probably code refactoring.  Actually we have 3
> different cases:
>
>  1) We don't use soft deleted records:
> 1.1) Do .delete() instead of .soft_delete()
> 1.2) Change query to avoid adding extra "deleted == 0" to each query
> 1.3) Drop "deleted" and "deleted_at" columns
>
>  2) We use soft deleted records for internal stuff "e.g. periodic tasks"
> 2.1) Refactor code somehow: E.g. store all required data by periodic task
> in some special table that has: (id, type, json_data) columns
> 2.2) On delete add record to this table
> 2.3-5) similar to 1.1, 1.2, 13
>
>  3) We use soft deleted records in API
> 3.1) Deprecated API call if it is possible
> 3.2) Make proxy call to ceilometer from API
> 3.3) On .delete() store info about records in (ceilometer, or somewhere
> else)
> 3.4-6) similar to 1.1, 1.2, 1.3
>
> This is not ready RoadMap, just base thoughts to start the constructive
> discussion in the mailing list, so %stacker% your opinion is very
> important!
>
>
>  Best regards,
> Boris Pavlovic
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Tim Bell]

If the deleted column is removed, how would the 'undelete' functionality be 
provided ? This saves operators when user accidents occur since restoring the 
whole database to a point in time affects the other tenants also.

Tim

> Hi all,
> 
> >>> I've never understood why we treat the DB as a LOG (keeping deleted == 0 
> >>> records around) when we should just use a LOG (or
> similar system) to begin with instead.
> 
> I can't agree more with you! Storing deleted records in tables is hardly 
> usable, bad for performance (as it makes tables and indexes
> larger) and it probably covers a very limited set of use cases (if
> any) of OpenStack users.
> 

If the deleted column is removed, how would the 'undelete' functionality be 
provided ? This saves operators when user accidents occur since restoring the 
whole database to a point in time affects the other tenants also.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Alexis Lee]

Roman Podoliaka said on Mon, Mar 10, 2014 at 03:04:06PM -0700:
> So we have a homework to do: find out what for projects use
> soft-deletes. I assume that soft-deletes are only used internally and
> aren't exposed to API users, but let's check that. At the same time
> all new projects should avoid using of soft-deletes from the start.

On that homework, deleted records can be interesting when aggregating
over time. For example, "nodes where over 100 instances went to ERROR
this month" or "nodes that hosted flavor  this month".

Operators might have written plugins to test these business concerns, so
although Ceilometer might be a better place to get that information, the
transition should be considered.


Alexis
-- 
Nova Engineer, HP Cloud.  AKA lealexis, lxsli.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Michael Still]

I see many examples in nova of where we still read rows with
read_deleted="yes". I think we need to see a plan for how to remove
all of those before we can progress this.

Michael

On Tue, Mar 11, 2014 at 9:06 AM, Johannes Erdfelt  wrote:
> On Mon, Mar 10, 2014, Joshua Harlow  wrote:
>> Sounds like a good idea to me.
>
> I generally think this is a good idea too.
>
>> I've never understood why we treat the DB as a LOG (keeping deleted == 0
>> records around) when we should just use a LOG (or similar system) to
>> begin with instead.
>>
>> Does anyone use the feature of switching deleted == 1 back to
>> deleted = 0? Has this worked out for u?
>
> This isn't the only potential use. It's possible that code depends on
> being able to still access deleted records.
>
> For instance, in the past we could delete an instance_type, but if an
> instance is still referencing it, code would still try to fetch it from
> the database some times.
>
> This particular example probably isn't an issue anymore since I think all
> of that has been moved to instance metadata specifically to avoid
> problems like this.
>
> That said, I think it's well worth the effort to simplify the code and
> make operators lives easier.
>
> JE
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Johannes Erdfelt]

On Mon, Mar 10, 2014, Joshua Harlow  wrote:
> Sounds like a good idea to me.

I generally think this is a good idea too.

> I've never understood why we treat the DB as a LOG (keeping deleted == 0
> records around) when we should just use a LOG (or similar system) to
> begin with instead.
> 
> Does anyone use the feature of switching deleted == 1 back to
> deleted = 0? Has this worked out for u?

This isn't the only potential use. It's possible that code depends on
being able to still access deleted records.

For instance, in the past we could delete an instance_type, but if an
instance is still referencing it, code would still try to fetch it from
the database some times.

This particular example probably isn't an issue anymore since I think all
of that has been moved to instance metadata specifically to avoid
problems like this.

That said, I think it's well worth the effort to simplify the code and
make operators lives easier.

JE


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Roman Podoliaka]

Hi all,

>>> I've never understood why we treat the DB as a LOG (keeping deleted == 0 
>>> records around) when we should just use a LOG (or similar system) to begin 
>>> with instead.

I can't agree more with you! Storing deleted records in tables is
hardly usable, bad for performance (as it makes tables and indexes
larger) and it probably covers a very limited set of use cases (if
any) of OpenStack users.

>>> One of approaches that I see is in step by step removing "deleted" column 
>>> from every table with probably code refactoring.

So we have a homework to do: find out what for projects use
soft-deletes. I assume that soft-deletes are only used internally and
aren't exposed to API users, but let's check that. At the same time
all new projects should avoid using of soft-deletes from the start.

Thanks,
Roman

On Mon, Mar 10, 2014 at 2:44 PM, Joshua Harlow  wrote:
> Sounds like a good idea to me.
>
> I've never understood why we treat the DB as a LOG (keeping deleted == 0
> records around) when we should just use a LOG (or similar system) to begin
> with instead.
>
> Does anyone use the feature of switching deleted == 1 back to deleted = 0?
> Has this worked out for u?
>
> Seems like some of the feedback on
> https://etherpad.openstack.org/p/operators-feedback-mar14 also suggests that
> this has been a operational pain-point for folks (Tool to delete things
> properly suggestions and such…).
>
> From: Boris Pavlovic 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
> 
> Date: Monday, March 10, 2014 at 1:29 PM
> To: OpenStack Development Mailing List ,
> Victor Sergeyev 
> Subject: [openstack-dev] [all][db][performance] Proposal: Get rid of soft
> deletion (step by step)
>
> Hi stackers,
>
> (It's proposal for Juno.)
>
> Intro:
>
> Soft deletion means that records from DB are not actually deleted, they are
> just marked as a "deleted". To mark record as a "deleted" we put in special
> table's column "deleted" record's ID value.
>
> Issue 1: Indexes & Queries
> We have to add in every query "AND deleted == 0" to get non-deleted records.
> It produce performance issue, cause we should add it in any index one
> "extra" column.
> As well it produce extra complexity in db migrations and building queries.
>
> Issue 2: Unique constraints
> Why we store ID in deleted and not True/False?
> The reason is that we would like to be able to create real DB unique
> constraints and avoid race conditions on "insert" operation.
>
> Sample: we Have table (id, name, password, deleted) we would like to put in
> column "name" only unique value.
>
> Approach without UC: if count(`select  where name = name`) == 0:
> insert(...)
> (race cause we are able to add new record between )
>
> Approach with UC: try: insert(...) except Duplicate: ...
>
> So to add UC we have to add them on (name, deleted). (to be able to make
> insert/delete/insert with same name)
>
> As well it produce performance issues, because we have to use Complex unique
> constraints on 2  or more columns. + extra code & complexity in db
> migrations.
>
> Issue 3: Garbage collector
>
> It is really hard to make garbage collector that will have good performance
> and be enough common to work in any case for any project.
> Without garbage collector DevOps have to cleanup records by hand, (risk to
> break something). If they don't cleanup DB they will get very soon
> performance issue.
>
> To put in a nutshell most important issues:
> 1) Extra complexity to each select query & extra column in each index
> 2) Extra column in each Unique Constraint (worse performance)
> 3) 2 Extra column in each table: (deleted, deleted_at)
> 4) Common garbage collector is required
>
>
> To resolve all these issues we should just remove soft deletion.
>
> One of approaches that I see is in step by step removing "deleted" column
> from every table with probably code refactoring.  Actually we have 3
> different cases:
>
> 1) We don't use soft deleted records:
> 1.1) Do .delete() instead of .soft_delete()
> 1.2) Change query to avoid adding extra "deleted == 0" to each query
> 1.3) Drop "deleted" and "deleted_at" columns
>
> 2) We use soft deleted records for internal stuff "e.g. periodic tasks"
> 2.1) Refactor code somehow: E.g. store all required data by periodic task in
> some special table that has: (id, type, json_data) columns
> 2.2) On delete add record to this table
> 2.3-5) similar to 1.1, 1.2, 13
>
> 3) We use soft deleted record

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Jay Pipes]

On Tue, 2014-03-11 at 01:29 +0400, Boris Pavlovic wrote:
> 
> To put in a nutshell most important issues:
> 1) Extra complexity to each select query & extra column in each index
> 2) Extra column in each Unique Constraint (worse performance)
> 3) 2 Extra column in each table: (deleted, deleted_at)
> 4) Common garbage collector is required

Nice summary of the problems related to soft deletion, Boris.

> To resolve all these issues we should just remove soft deletion.
> 
> One of approaches that I see is in step by step removing "deleted"
> column from every table with probably code refactoring.  Actually we
> have 3 different cases:
> 
> 1) We don't use soft deleted records: 
> 1.1) Do .delete() instead of .soft_delete()
> 1.2) Change query to avoid adding extra "deleted == 0" to each query
> 1.3) Drop "deleted" and "deleted_at" columns
> 
> 2) We use soft deleted records for internal stuff "e.g. periodic
> tasks"
> 2.1) Refactor code somehow: E.g. store all required data by periodic
> task in some special table that has: (id, type, json_data) columns
> 2.2) On delete add record to this table 
> 2.3-5) similar to 1.1, 1.2, 13
> 
> 3) We use soft deleted records in API 
> 3.1) Deprecated API call if it is possible 
> 3.2) Make proxy call to ceilometer from API 
> 3.3) On .delete() store info about records in (ceilometer, or
> somewhere else) 
> 3.4-6) similar to 1.1, 1.2, 1.3

I would actually prefer this "solution", at least for server instances:

1. Remove any contractual obligation in the API to allow servers with
the same "name" to exist, as long as only one of those servers is not
deleted. As I've mentioned before, I think this is exceedingly silly to
slow down the operation of Nova just to allow a user to create a server,
delete it, and immediately create a server with the same name.

2. Make the unique constraint for the server name be on (project_id,
name) and be done with it.

3. Remove deleted and deleted_at from the instances table.

4. Don't allow any delete() operation at all on the
nova.objects.instance object at all.

3. Hard delete records from the instances table on a periodic basis
using an external archiver that either just deletes the records in
instances that are in ERROR or TERMINATED vm_state (as is possible if
ceilometer is providing your bookkeeping) or move those records into an
archival table (as would be necessary if you are not running Ceilometer
and need some history of these things).

For other objects in the system, I think your solution #1 would work
fine.

Best,
-jay



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Joshua Harlow]

Sounds like a good idea to me.

I've never understood why we treat the DB as a LOG (keeping deleted == 0 
records around) when we should just use a LOG (or similar system) to begin with 
instead.

Does anyone use the feature of switching deleted == 1 back to deleted = 0? Has 
this worked out for u?

Seems like some of the feedback on 
https://etherpad.openstack.org/p/operators-feedback-mar14 also suggests that 
this has been a operational pain-point for folks (Tool to delete things 
properly suggestions and such…).

From: Boris Pavlovic mailto:bpavlo...@mirantis.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date: Monday, March 10, 2014 at 1:29 PM
To: OpenStack Development Mailing List 
mailto:openstack-dev@lists.openstack.org>>, 
Victor Sergeyev mailto:vserge...@mirantis.com>>
Subject: [openstack-dev] [all][db][performance] Proposal: Get rid of soft 
deletion (step by step)

Hi stackers,

(It's proposal for Juno.)

Intro:

Soft deletion means that records from DB are not actually deleted, they are 
just marked as a "deleted". To mark record as a "deleted" we put in special 
table's column "deleted" record's ID value.

Issue 1: Indexes & Queries
We have to add in every query "AND deleted == 0" to get non-deleted records.
It produce performance issue, cause we should add it in any index one "extra" 
column.
As well it produce extra complexity in db migrations and building queries.

Issue 2: Unique constraints
Why we store ID in deleted and not True/False?
The reason is that we would like to be able to create real DB unique 
constraints and avoid race conditions on "insert" operation.

Sample: we Have table (id, name, password, deleted) we would like to put in 
column "name" only unique value.

Approach without UC: if count(`select  where name = name`) == 0: insert(...)
(race cause we are able to add new record between )

Approach with UC: try: insert(...) except Duplicate: ...

So to add UC we have to add them on (name, deleted). (to be able to make 
insert/delete/insert with same name)

As well it produce performance issues, because we have to use Complex unique 
constraints on 2  or more columns. + extra code & complexity in db migrations.

Issue 3: Garbage collector

It is really hard to make garbage collector that will have good performance and 
be enough common to work in any case for any project.
Without garbage collector DevOps have to cleanup records by hand, (risk to 
break something). If they don't cleanup DB they will get very soon performance 
issue.

To put in a nutshell most important issues:
1) Extra complexity to each select query & extra column in each index
2) Extra column in each Unique Constraint (worse performance)
3) 2 Extra column in each table: (deleted, deleted_at)
4) Common garbage collector is required


To resolve all these issues we should just remove soft deletion.

One of approaches that I see is in step by step removing "deleted" column from 
every table with probably code refactoring.  Actually we have 3 different cases:

1) We don't use soft deleted records:
1.1) Do .delete() instead of .soft_delete()
1.2) Change query to avoid adding extra "deleted == 0" to each query
1.3) Drop "deleted" and "deleted_at" columns

2) We use soft deleted records for internal stuff "e.g. periodic tasks"
2.1) Refactor code somehow: E.g. store all required data by periodic task in 
some special table that has: (id, type, json_data) columns
2.2) On delete add record to this table
2.3-5) similar to 1.1, 1.2, 13

3) We use soft deleted records in API
3.1) Deprecated API call if it is possible
3.2) Make proxy call to ceilometer from API
3.3) On .delete() store info about records in (ceilometer, or somewhere else)
3.4-6) similar to 1.1, 1.2, 1.3

This is not ready RoadMap, just base thoughts to start the constructive 
discussion in the mailing list, so %stacker% your opinion is very important!


Best regards,
Boris Pavlovic
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step)

[Boris Pavlovic]

Hi stackers,

(It's proposal for Juno.)

Intro:

Soft deletion means that records from DB are not actually deleted, they are
just marked as a "deleted". To mark record as a "deleted" we put in special
table's column "deleted" record's ID value.

Issue 1: Indexes & Queries
We have to add in every query "AND deleted == 0" to get non-deleted records.
It produce performance issue, cause we should add it in any index one
"extra" column.
As well it produce extra complexity in db migrations and building queries.

Issue 2: Unique constraints
Why we store ID in deleted and not True/False?
The reason is that we would like to be able to create real DB unique
constraints and avoid race conditions on "insert" operation.

Sample: we Have table (id, name, password, deleted) we would like to put in
column "name" only unique value.

Approach without UC: if count(`select  where name = name`) == 0:
insert(...)
(race cause we are able to add new record between )

Approach with UC: try: insert(...) except Duplicate: ...

So to add UC we have to add them on (name, deleted). (to be able to make
insert/delete/insert with same name)

As well it produce performance issues, because we have to use Complex
unique constraints on 2  or more columns. + extra code & complexity in db
migrations.

Issue 3: Garbage collector

It is really hard to make garbage collector that will have good performance
and be enough common to work in any case for any project.
Without garbage collector DevOps have to cleanup records by hand, (risk to
break something). If they don't cleanup DB they will get very soon
performance issue.

To put in a nutshell most important issues:
1) Extra complexity to each select query & extra column in each index
2) Extra column in each Unique Constraint (worse performance)
3) 2 Extra column in each table: (deleted, deleted_at)
4) Common garbage collector is required


To resolve all these issues we should just remove soft deletion.

One of approaches that I see is in step by step removing "deleted" column
from every table with probably code refactoring.  Actually we have 3
different cases:

1) We don't use soft deleted records:
1.1) Do .delete() instead of .soft_delete()
1.2) Change query to avoid adding extra "deleted == 0" to each query
1.3) Drop "deleted" and "deleted_at" columns

2) We use soft deleted records for internal stuff "e.g. periodic tasks"
2.1) Refactor code somehow: E.g. store all required data by periodic task
in some special table that has: (id, type, json_data) columns
2.2) On delete add record to this table
2.3-5) similar to 1.1, 1.2, 13

3) We use soft deleted records in API
3.1) Deprecated API call if it is possible
3.2) Make proxy call to ceilometer from API
3.3) On .delete() store info about records in (ceilometer, or somewhere
else)
3.4-6) similar to 1.1, 1.2, 1.3

This is not ready RoadMap, just base thoughts to start the constructive
discussion in the mailing list, so %stacker% your opinion is very
important!


Best regards,
Boris Pavlovic
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev