Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
On 17 March 2014 20:54, Jay Pipes wrote: > After hearing from Tim Bell and others, I think that a two-pronged > approach is most useful. First, the above-mentioned UI changes to > prevent common mistakes, and second, using a consistent, standardized > way of undoing certain operations (Boris' proposal). > I'm very much in favor of this proposal. The primary defense against end-user mistakes should be within Horizon - where currently the 'Reboot' and 'Terminate' buttons for an instance are perilously close to each other, among other issues. There are already UI improvement activities in play to help with this, but the UX team could probably do with more input. The CLI, in my view, is less prone to 'finger trouble' and is perhaps less worthy of attention in this regard. A secondary defense is some sort of systematic protection against actions like deleting an image from Glance when it's still used by an instance, thus preventing actions like an instance migration. In a situation like this, soft deletion isn't exactly the right answer - but instead perhaps the right method is to remove the image from general visibility until such time that the instance(s) using it are no longer around. Then the image is actually removed from the registry. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
Interesting proposal... there would also be a benefit of different tables per program from an operational perspective. If I need to recover a database for any reason, having different tables would ensure that I could restore glance to a point in time without having to lose the nova delete data. Tim From: Mark Washenberger [mailto:mark.washenber...@markwash.net] Sent: 17 March 2014 21:08 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources On Thu, Mar 13, 2014 at 12:42 PM, Boris Pavlovic mailto:bpavlo...@mirantis.com>> wrote: Hi stackers, As a result of discussion: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step) http://osdir.com/ml/openstack-dev/2014-03/msg00947.html I understood that there should be another proposal. About how we should implement Restorable & Delayed Deletion of OpenStack Resource in common way & without these hacks with soft deletion in DB. It is actually very simple, take a look at this document: https://docs.google.com/document/d/1WGrIgMtWJqPDyT6PkPeZhNpej2Q9Mwimula8S8lYGV4/edit?usp=sharing Best regards, Boris Pavlovic ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Boris, Before I voice a little disagreement, I'd like to thank you for kicking off this discussion and stress that I strongly agree with your view (pulled from the other thread) > To put in a nutshell: Restoring Delete resources / Delayed Deletion != Soft > deletion. This is absolutely correct and the key to unlocking the problem we have. However, because of migrations and because being explicit is better than being implicit, I disagree about the idea of lumping deleted resources all into the same table. For glance, I'd much rather have a table "deleted_images" than a table "deleted_resources" that has some image entries. There are a number of reasons, I'll try to give a quick high-level view of them. 1) Migrations for deleted data are more straightforward and more obviously necessary. 2) It is possible to make specific modifications to the "deleted_X" schema. 3) It is possible to take many tables that are used to represent a single active resource (images, image_locations, image_tags, image_properties) and combine them into a single table for a deleted resource. This is actually super important as today we have the problem of not always knowing what image_properties were actually deleted prior to the image deletion vs the ones that were deleted as a part of the image deletion. 4) It makes it a conscious choice to decide to have certain types of resources restorable or have delayed deletes. As you said before, many types of resources just don't need this functionality, so let's not make it a feature of the common base class. (I am assuming for #2 and #3 that this common approach would be implemented something like "deleted_resource['data'] = json.dumps(dict(active_resource))", sorry if that is seriously incorrect.) Thanks for your consideration, markwash ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
On Thu, Mar 13, 2014 at 12:42 PM, Boris Pavlovic wrote: > Hi stackers, > > As a result of discussion: > [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion > (step by step) > http://osdir.com/ml/openstack-dev/2014-03/msg00947.html > > I understood that there should be another proposal. About how we should > implement Restorable & Delayed Deletion of OpenStack Resource in common way > & without these hacks with soft deletion in DB. It is actually very > simple, take a look at this document: > > > https://docs.google.com/document/d/1WGrIgMtWJqPDyT6PkPeZhNpej2Q9Mwimula8S8lYGV4/edit?usp=sharing > > > Best regards, > Boris Pavlovic > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > Hi Boris, Before I voice a little disagreement, I'd like to thank you for kicking off this discussion and stress that I strongly agree with your view (pulled from the other thread) > To put in a nutshell: Restoring Delete resources / Delayed Deletion != Soft deletion. This is absolutely correct and the key to unlocking the problem we have. However, because of migrations and because being explicit is better than being implicit, I disagree about the idea of lumping deleted resources all into the same table. For glance, I'd much rather have a table "deleted_images" than a table "deleted_resources" that has some image entries. There are a number of reasons, I'll try to give a quick high-level view of them. 1) Migrations for deleted data are more straightforward and more obviously necessary. 2) It is possible to make specific modifications to the "deleted_X" schema. 3) It is possible to take many tables that are used to represent a single active resource (images, image_locations, image_tags, image_properties) and combine them into a single table for a deleted resource. This is actually super important as today we have the problem of not always knowing what image_properties were actually deleted prior to the image deletion vs the ones that were deleted as a part of the image deletion. 4) It makes it a conscious choice to decide to have certain types of resources restorable or have delayed deletes. As you said before, many types of resources just don't need this functionality, so let's not make it a feature of the common base class. (I am assuming for #2 and #3 that this common approach would be implemented something like "deleted_resource['data'] = json.dumps(dict(active_resource))", sorry if that is seriously incorrect.) Thanks for your consideration, markwash ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
On Sun, 2014-03-16 at 23:02 -0700, Allamaraju, Subbu wrote: > Hi Boris, > > I just read the other thread. As Jay asked in [1], have you considered > precautions in the UI in stead? That should take care of mistakes with manual > deletes. > > Thx > Subbu > > [1] http://lists.openstack.org/pipermail/openstack-dev/2014-March/029784.html After hearing from Tim Bell and others, I think that a two-pronged approach is most useful. First, the above-mentioned UI changes to prevent common mistakes, and second, using a consistent, standardized way of undoing certain operations (Boris' proposal). Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
Hi Boris, I just read the other thread. As Jay asked in [1], have you considered precautions in the UI in stead? That should take care of mistakes with manual deletes. Thx Subbu [1] http://lists.openstack.org/pipermail/openstack-dev/2014-March/029784.html On Mar 16, 2014, at 10:37 PM, Boris Pavlovic wrote: > Subbu, > > No it's not too late. It's just proposal for Juno. > > First of all, you should keep in mind that in you cause it's probably > automated. In case of Web Hosting it's done by end users (so it's not > automated). > > If you spend some time and read discussion about removing "Soft deletion" [1] > you'll see that restoring "deleted" stuff is popular thing. So the goal of > this proposal is to make standard approach for restoring that won't use "soft > deletion". > > > [1] http://osdir.com/ml/openstack-dev/2014-03/msg00947.html > > Best regards, > Boris Pavlovic > > > On Mon, Mar 17, 2014 at 9:23 AM, Allamaraju, Subbu wrote: > Hope this is not too late to ask this question, but isn't this extra code > just fat finger mistakes? > > IME, most provisioning on cloud happens via automated tools, and it seems > counter-productive to design a feature for manual operations. > > Thx, > Subbu > > On Mar 13, 2014, at 12:42 PM, Boris Pavlovic wrote: > > > Hi stackers, > > > > As a result of discussion: > > [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion > > (step by step) > > http://osdir.com/ml/openstack-dev/2014-03/msg00947.html > > > > I understood that there should be another proposal. About how we should > > implement Restorable & Delayed Deletion of OpenStack Resource in common way > > & without these hacks with soft deletion in DB. It is actually very > > simple, take a look at this document: > > > > https://docs.google.com/document/d/1WGrIgMtWJqPDyT6PkPeZhNpej2Q9Mwimula8S8lYGV4/edit?usp=sharing > > > > > > Best regards, > > Boris Pavlovic > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
Subbu, No it's not too late. It's just proposal for Juno. First of all, you should keep in mind that in you cause it's probably automated. In case of Web Hosting it's done by end users (so it's not automated). If you spend some time and read discussion about removing "Soft deletion" [1] you'll see that restoring "deleted" stuff is popular thing. So the goal of this proposal is to make standard approach for restoring that won't use "soft deletion". [1] http://osdir.com/ml/openstack-dev/2014-03/msg00947.html Best regards, Boris Pavlovic On Mon, Mar 17, 2014 at 9:23 AM, Allamaraju, Subbu wrote: > Hope this is not too late to ask this question, but isn't this extra code > just fat finger mistakes? > > IME, most provisioning on cloud happens via automated tools, and it seems > counter-productive to design a feature for manual operations. > > Thx, > Subbu > > On Mar 13, 2014, at 12:42 PM, Boris Pavlovic > wrote: > > > Hi stackers, > > > > As a result of discussion: > > [openstack-dev] [all][db][performance] Proposal: Get rid of soft > deletion (step by step) > > http://osdir.com/ml/openstack-dev/2014-03/msg00947.html > > > > I understood that there should be another proposal. About how we should > implement Restorable & Delayed Deletion of OpenStack Resource in common way > & without these hacks with soft deletion in DB. It is actually very > simple, take a look at this document: > > > > > https://docs.google.com/document/d/1WGrIgMtWJqPDyT6PkPeZhNpej2Q9Mwimula8S8lYGV4/edit?usp=sharing > > > > > > Best regards, > > Boris Pavlovic > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
Hope this is not too late to ask this question, but isn't this extra code just fat finger mistakes? IME, most provisioning on cloud happens via automated tools, and it seems counter-productive to design a feature for manual operations. Thx, Subbu On Mar 13, 2014, at 12:42 PM, Boris Pavlovic wrote: > Hi stackers, > > As a result of discussion: > [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion > (step by step) > http://osdir.com/ml/openstack-dev/2014-03/msg00947.html > > I understood that there should be another proposal. About how we should > implement Restorable & Delayed Deletion of OpenStack Resource in common way & > without these hacks with soft deletion in DB. It is actually very simple, > take a look at this document: > > https://docs.google.com/document/d/1WGrIgMtWJqPDyT6PkPeZhNpej2Q9Mwimula8S8lYGV4/edit?usp=sharing > > > Best regards, > Boris Pavlovic > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
Interesting, the usage of stop to do this could work since the semantics are similar enough. People could even remap delete to stop and force-delete to delete and this would be a solution as well (or as u said use a cronjob/deletion service/janitor crew that does the force-delete). Good idea! Sent from my really tiny device... > On Mar 15, 2014, at 8:41 AM, "Clint Byrum" wrote: > > Excerpts from Joshua Harlow's message of 2014-03-13 16:16:50 -0700: >> Seems ok to me, and likely a good start, although I'm still not very >> comfortable with the effects of soft_deletion (unless its done by admins >> only), to me it complicates scheduling (can u schedule to something that has >> been soft_deleted, likely not). It also creates a pool of resources that >> can't be used but can't be deleted either, that sounds a little bad and >> wastes companies $$ and it reinforces non-cloudly concepts. It also seems >> very complex, especially when your start connecting more and more resources >> together via heat or other system (the whole graph of resources now must be >> soft_deleted, wasting more $$, and how does one restore the graph of >> resources if some of them were also hard_deleted). > > I think you stay clear of scheduling if you treat it as a stopped > resource. It is costing you to be there, even if it isn't using the CPU > and RAM, it is a form of reservation. > > The pool of unusable resources must be built into the price for > undeletable resources. How long to keep things around is a business > decision. I could see an evolution of the feature that includes undelete > period in the flavor definition. > > The fact that one would need to be able to undelete whole applications > is just a missing feature. In the case of Heat, it would definitely get > complicated if you went out of band and accidentally deleted things but > it would be uncomplicated as long as you undeleted it before Heat tried > to do an in-place operation like a Metadata change + waitcondition or a > rebuild/resize. > > I think though, that we already have this feature for most things in the > form of "stop" versus "delete". The way I would implement undelete is at > the policy level. > > Deny delete to users, and provide a cron-like functionality for > deleting. Let them decide how long they'd like to keep things around for, > and then let the cron-like thing do the actual deleting. I believe a few > of these cron-as-a-service things already exist. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
Excerpts from Joshua Harlow's message of 2014-03-13 16:16:50 -0700: > Seems ok to me, and likely a good start, although I'm still not very > comfortable with the effects of soft_deletion (unless its done by admins > only), to me it complicates scheduling (can u schedule to something that has > been soft_deleted, likely not). It also creates a pool of resources that > can't be used but can't be deleted either, that sounds a little bad and > wastes companies $$ and it reinforces non-cloudly concepts. It also seems > very complex, especially when your start connecting more and more resources > together via heat or other system (the whole graph of resources now must be > soft_deleted, wasting more $$, and how does one restore the graph of > resources if some of them were also hard_deleted). > I think you stay clear of scheduling if you treat it as a stopped resource. It is costing you to be there, even if it isn't using the CPU and RAM, it is a form of reservation. The pool of unusable resources must be built into the price for undeletable resources. How long to keep things around is a business decision. I could see an evolution of the feature that includes undelete period in the flavor definition. The fact that one would need to be able to undelete whole applications is just a missing feature. In the case of Heat, it would definitely get complicated if you went out of band and accidentally deleted things but it would be uncomplicated as long as you undeleted it before Heat tried to do an in-place operation like a Metadata change + waitcondition or a rebuild/resize. I think though, that we already have this feature for most things in the form of "stop" versus "delete". The way I would implement undelete is at the policy level. Deny delete to users, and provide a cron-like functionality for deleting. Let them decide how long they'd like to keep things around for, and then let the cron-like thing do the actual deleting. I believe a few of these cron-as-a-service things already exist. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
+1 .. looks like it would cover the accidental use case. Something could then be included into oslo for standardisation. Tim From: Boris Pavlovic [mailto:bpavlo...@mirantis.com] Sent: 13 March 2014 20:42 To: OpenStack Development Mailing List Subject: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources Hi stackers, As a result of discussion: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step) http://osdir.com/ml/openstack-dev/2014-03/msg00947.html I understood that there should be another proposal. About how we should implement Restorable & Delayed Deletion of OpenStack Resource in common way & without these hacks with soft deletion in DB. It is actually very simple, take a look at this document: https://docs.google.com/document/d/1WGrIgMtWJqPDyT6PkPeZhNpej2Q9Mwimula8S8lYGV4/edit?usp=sharing Best regards, Boris Pavlovic ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
Seems ok to me, and likely a good start, although I'm still not very comfortable with the effects of soft_deletion (unless its done by admins only), to me it complicates scheduling (can u schedule to something that has been soft_deleted, likely not). It also creates a pool of resources that can't be used but can't be deleted either, that sounds a little bad and wastes companies $$ and it reinforces non-cloudly concepts. It also seems very complex, especially when your start connecting more and more resources together via heat or other system (the whole graph of resources now must be soft_deleted, wasting more $$, and how does one restore the graph of resources if some of them were also hard_deleted). -Josh From: Mike Wilson mailto:geekinu...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Thursday, March 13, 2014 at 1:26 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources After a read through seems pretty good. +1 On Thu, Mar 13, 2014 at 1:42 PM, Boris Pavlovic mailto:bpavlo...@mirantis.com>> wrote: Hi stackers, As a result of discussion: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step) http://osdir.com/ml/openstack-dev/2014-03/msg00947.html I understood that there should be another proposal. About how we should implement Restorable & Delayed Deletion of OpenStack Resource in common way & without these hacks with soft deletion in DB. It is actually very simple, take a look at this document: https://docs.google.com/document/d/1WGrIgMtWJqPDyT6PkPeZhNpej2Q9Mwimula8S8lYGV4/edit?usp=sharing Best regards, Boris Pavlovic ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
After a read through seems pretty good. +1 On Thu, Mar 13, 2014 at 1:42 PM, Boris Pavlovic wrote: > Hi stackers, > > As a result of discussion: > [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion > (step by step) > http://osdir.com/ml/openstack-dev/2014-03/msg00947.html > > I understood that there should be another proposal. About how we should > implement Restorable & Delayed Deletion of OpenStack Resource in common way > & without these hacks with soft deletion in DB. It is actually very > simple, take a look at this document: > > > https://docs.google.com/document/d/1WGrIgMtWJqPDyT6PkPeZhNpej2Q9Mwimula8S8lYGV4/edit?usp=sharing > > > Best regards, > Boris Pavlovic > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [db][all] (Proposal) Restorable & Delayed deletion of OS Resources
Hi stackers, As a result of discussion: [openstack-dev] [all][db][performance] Proposal: Get rid of soft deletion (step by step) http://osdir.com/ml/openstack-dev/2014-03/msg00947.html I understood that there should be another proposal. About how we should implement Restorable & Delayed Deletion of OpenStack Resource in common way & without these hacks with soft deletion in DB. It is actually very simple, take a look at this document: https://docs.google.com/document/d/1WGrIgMtWJqPDyT6PkPeZhNpej2Q9Mwimula8S8lYGV4/edit?usp=sharing Best regards, Boris Pavlovic ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
