Re: [openstack-dev] [nova] How were we going to remove soft delete again?

2015-11-30 Thread Alex Xu 
2015-11-30 19:42 GMT+08:00 Sean Dague :

> On 11/24/2015 11:36 AM, Matt Riedemann wrote:
> > I know in Vancouver we talked about no more soft delete and in Mitaka
> > lxsli decoupled the nova models from the SoftDeleteMixin [1].
> >
> > From what I remember, the idea is to not add the deleted column to new
> > tables, to not expose soft deleted resources in the REST API in new
> > ways, and to eventually drop the deleted column from the models.
> >
> > I bring up the REST API because I was tinkering with the idea of
> > allowing non-admins to list/show their (soft) deleted instances [2].
> > Doing that, however, would expose more of the REST API to deleted
> > resources which makes it harder to remove from the data model.
> >
> > My question is, how were we thinking we were going to remove the deleted
> > column from the data model in a backward compatible way? A new
> > microversion in the REST API isn't going to magically work if we drop
> > the column in the data model, since anything before that microversion
> > should still work - like listing deleted instances for the admin.
> >
> > Am I forgetting something? There were a lot of ideas going around the
> > room during the session in Vancouver and I'd like to sort out the
> > eventual long-term plan so we can document it in the devref about
> > policies so that when ideas like [2] come up we can point to the policy
> > and say 'no we aren't going to do that and here's why'.
> >
> > [1]
> >
> http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/no-more-soft-delete.html
> >
> > [2]
> >
> https://blueprints.launchpad.net/nova/+spec/non-admin-list-deleted-instances
> >
> >
>
> On the API compat side. There is no contract with the user for how long
> deleted items will be shown, or how old the deleted history is.
>
> The contract semantics are:
>
> * by default - don't show me anything that's deleted
> * with a flag - ok, you can *also* show me deleted things
>
> The one places soft deleted removal is going to be semantically
> meaningful is on GET /servers/details?changes-since=
>
> Because in that case it's used by most people as a journal, and includes
> deleted instances. It's our janky event stream for servers (given that
> we have no other real event stream). That would need something better
> before we make that query not possible.
>
> I also agree, it's not clear to me why removing soft delete off of
> things like instances is hugely critical. I think getting rid of it as
> default for new things is good. But some places it's pretty intrinsic to
> the information people pull out.
>

+1, for user to check out the history, query the deleted instance sounds
like useful.

For showing the deleted instance for non-admin user, looks like it's hard
for operator to make promise to user that how long the deleted instance can
be queried anymore. Currently the db archive just archive all the entries,
whatever it was deleted long time ago, or just in seconds. We probably can
add some filter to db archive call. But anyway this is another thing. Also
want to figure out why query deleted instance is useless first.


>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How were we going to remove soft delete again?

2015-11-27 Thread Matt Riedemann 



On 11/26/2015 6:17 AM, John Garbutt wrote:

On 26 November 2015 at 12:10, John Garbutt  wrote:

On 24 November 2015 at 16:36, Matt Riedemann  wrote:

I know in Vancouver we talked about no more soft delete and in Mitaka lxsli
decoupled the nova models from the SoftDeleteMixin [1].

 From what I remember, the idea is to not add the deleted column to new
tables, to not expose soft deleted resources in the REST API in new ways,
and to eventually drop the deleted column from the models.

I bring up the REST API because I was tinkering with the idea of allowing
non-admins to list/show their (soft) deleted instances [2]. Doing that,
however, would expose more of the REST API to deleted resources which makes
it harder to remove from the data model.

My question is, how were we thinking we were going to remove the deleted
column from the data model in a backward compatible way? A new microversion
in the REST API isn't going to magically work if we drop the column in the
data model, since anything before that microversion should still work - like
listing deleted instances for the admin.

Am I forgetting something? There were a lot of ideas going around the room
during the session in Vancouver and I'd like to sort out the eventual
long-term plan so we can document it in the devref about policies so that
when ideas like [2] come up we can point to the policy and say 'no we aren't
going to do that and here's why'.

[1]
http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/no-more-soft-delete.html
[2]
https://blueprints.launchpad.net/nova/+spec/non-admin-list-deleted-instances


 From my memory, step 1 is to ensure we don't keep adding soft delete
by default/accident, which is where the explicit mix-in should help.

Step 2, is removing existing soft_deletes. Now we can add a new
microversion to remove the concept of requesting deleted things, but
as you point out, that doesn't help the older microversions.

What we could raise 403 errors when users request deleted things in
older versions of the API. I don't like that breaking API change, but
I also don't like the idea of keeping soft_delete in the database for
ever. Its a the case of picking the best of two bad outcomes. I am not
sure we have reached consensus on the preferred approach yet.


I just realised, my text is ambiguous...

There is a difference between soft deleted instances, and soft delete in the DB.

If the instance could still be restored, and is not yet deleted, it
makes sense that policy could allow a non-admin to see those. But
thats a non-db-deleted instance in the SOFT_DELETED state.

I am still leaning towards killing the APIs that allow you to read in
DB soft-deleted data. Although, in some ways thats because the API
changes based on the DB retention policy of the deployer, which seems
very odd.

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



What is the main reason again for removing soft deleted (deleted != 0) 
resources? Because of DB bloat? If that's the case, aren't operators 
pretty used to archiving/purging by now? Granted, the in-tree archive 
command is broken, and I'm working on fixing that, and we have a change 
up to add a purge command for (soft) deleted instances, but I'm trying 
to see if I'm forgetting something else here.


It's like soft delete is a Frankenstein and we're a mob out to kill it, 
but I forget every 6 months exactly why, and if it's totally necessary 
(and worth backward incompatible API changes). I think that's a question 
that operators and users should really answer rather than the nova dev team.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How were we going to remove soft delete again?

2015-11-26 Thread John Garbutt 
On 24 November 2015 at 16:36, Matt Riedemann  wrote:
> I know in Vancouver we talked about no more soft delete and in Mitaka lxsli
> decoupled the nova models from the SoftDeleteMixin [1].
>
> From what I remember, the idea is to not add the deleted column to new
> tables, to not expose soft deleted resources in the REST API in new ways,
> and to eventually drop the deleted column from the models.
>
> I bring up the REST API because I was tinkering with the idea of allowing
> non-admins to list/show their (soft) deleted instances [2]. Doing that,
> however, would expose more of the REST API to deleted resources which makes
> it harder to remove from the data model.
>
> My question is, how were we thinking we were going to remove the deleted
> column from the data model in a backward compatible way? A new microversion
> in the REST API isn't going to magically work if we drop the column in the
> data model, since anything before that microversion should still work - like
> listing deleted instances for the admin.
>
> Am I forgetting something? There were a lot of ideas going around the room
> during the session in Vancouver and I'd like to sort out the eventual
> long-term plan so we can document it in the devref about policies so that
> when ideas like [2] come up we can point to the policy and say 'no we aren't
> going to do that and here's why'.
>
> [1]
> http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/no-more-soft-delete.html
> [2]
> https://blueprints.launchpad.net/nova/+spec/non-admin-list-deleted-instances

>From my memory, step 1 is to ensure we don't keep adding soft delete
by default/accident, which is where the explicit mix-in should help.

Step 2, is removing existing soft_deletes. Now we can add a new
microversion to remove the concept of requesting deleted things, but
as you point out, that doesn't help the older microversions.

What we could raise 403 errors when users request deleted things in
older versions of the API. I don't like that breaking API change, but
I also don't like the idea of keeping soft_delete in the database for
ever. Its a the case of picking the best of two bad outcomes. I am not
sure we have reached consensus on the preferred approach yet.

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] How were we going to remove soft delete again?

2015-11-26 Thread John Garbutt 
On 26 November 2015 at 12:10, John Garbutt  wrote:
> On 24 November 2015 at 16:36, Matt Riedemann  
> wrote:
>> I know in Vancouver we talked about no more soft delete and in Mitaka lxsli
>> decoupled the nova models from the SoftDeleteMixin [1].
>>
>> From what I remember, the idea is to not add the deleted column to new
>> tables, to not expose soft deleted resources in the REST API in new ways,
>> and to eventually drop the deleted column from the models.
>>
>> I bring up the REST API because I was tinkering with the idea of allowing
>> non-admins to list/show their (soft) deleted instances [2]. Doing that,
>> however, would expose more of the REST API to deleted resources which makes
>> it harder to remove from the data model.
>>
>> My question is, how were we thinking we were going to remove the deleted
>> column from the data model in a backward compatible way? A new microversion
>> in the REST API isn't going to magically work if we drop the column in the
>> data model, since anything before that microversion should still work - like
>> listing deleted instances for the admin.
>>
>> Am I forgetting something? There were a lot of ideas going around the room
>> during the session in Vancouver and I'd like to sort out the eventual
>> long-term plan so we can document it in the devref about policies so that
>> when ideas like [2] come up we can point to the policy and say 'no we aren't
>> going to do that and here's why'.
>>
>> [1]
>> http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/no-more-soft-delete.html
>> [2]
>> https://blueprints.launchpad.net/nova/+spec/non-admin-list-deleted-instances
>
> From my memory, step 1 is to ensure we don't keep adding soft delete
> by default/accident, which is where the explicit mix-in should help.
>
> Step 2, is removing existing soft_deletes. Now we can add a new
> microversion to remove the concept of requesting deleted things, but
> as you point out, that doesn't help the older microversions.
>
> What we could raise 403 errors when users request deleted things in
> older versions of the API. I don't like that breaking API change, but
> I also don't like the idea of keeping soft_delete in the database for
> ever. Its a the case of picking the best of two bad outcomes. I am not
> sure we have reached consensus on the preferred approach yet.

I just realised, my text is ambiguous...

There is a difference between soft deleted instances, and soft delete in the DB.

If the instance could still be restored, and is not yet deleted, it
makes sense that policy could allow a non-admin to see those. But
thats a non-db-deleted instance in the SOFT_DELETED state.

I am still leaning towards killing the APIs that allow you to read in
DB soft-deleted data. Although, in some ways thats because the API
changes based on the DB retention policy of the deployer, which seems
very odd.

Thanks,
johnthetubaguy

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] How were we going to remove soft delete again?

2015-11-24 Thread Matt Riedemann 
I know in Vancouver we talked about no more soft delete and in Mitaka 
lxsli decoupled the nova models from the SoftDeleteMixin [1].


From what I remember, the idea is to not add the deleted column to new 
tables, to not expose soft deleted resources in the REST API in new 
ways, and to eventually drop the deleted column from the models.


I bring up the REST API because I was tinkering with the idea of 
allowing non-admins to list/show their (soft) deleted instances [2]. 
Doing that, however, would expose more of the REST API to deleted 
resources which makes it harder to remove from the data model.


My question is, how were we thinking we were going to remove the deleted 
column from the data model in a backward compatible way? A new 
microversion in the REST API isn't going to magically work if we drop 
the column in the data model, since anything before that microversion 
should still work - like listing deleted instances for the admin.


Am I forgetting something? There were a lot of ideas going around the 
room during the session in Vancouver and I'd like to sort out the 
eventual long-term plan so we can document it in the devref about 
policies so that when ideas like [2] come up we can point to the policy 
and say 'no we aren't going to do that and here's why'.


[1] 
http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/no-more-soft-delete.html
[2] 
https://blueprints.launchpad.net/nova/+spec/non-admin-list-deleted-instances


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev