Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Sean Dague]

On 11/17/2015 10:51 PM, Matt Riedemann wrote:

> 
> I *don't* see any DB APIs for deleting instance actions.
> 
> Kind of an important difference there.  Jay got it at least. :)
> 
>>
>> Were we just planning on instance_actions living forever in the database?
>>
>> Should we soft delete instance_actions when we delete the referenced
>> instance?
>>
>> Or should we (hard) delete instance_actions when we archive (move to
>> shadow tables) soft deleted instances?
>>
>> This is going to be a blocker to getting nova-manage db
>> archive_deleted_rows working.
>>
>> [1] https://review.openstack.org/#/c/246635/

instance_actions seems extremely useful, and at the ops meetups I've
been to has been one of the favorite features because it allows and easy
interface for "going back in time" to figure out what happened.

I'd suggest the following:

1. soft deleting and instance does nothing with instance actions.

2. archiving instance (soft delete -> actually deleted) also archives
off instance actions.

3. update instance_actions API so that you can get instance_actions for
deleted instances (which I think doesn't work today).

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Matt Riedemann]

On 11/20/2015 8:18 AM, Sean Dague wrote:

On 11/17/2015 10:51 PM, Matt Riedemann wrote:



I *don't* see any DB APIs for deleting instance actions.

Kind of an important difference there.  Jay got it at least. :)



Were we just planning on instance_actions living forever in the database?

Should we soft delete instance_actions when we delete the referenced
instance?

Or should we (hard) delete instance_actions when we archive (move to
shadow tables) soft deleted instances?

This is going to be a blocker to getting nova-manage db
archive_deleted_rows working.

[1] https://review.openstack.org/#/c/246635/


instance_actions seems extremely useful, and at the ops meetups I've
been to has been one of the favorite features because it allows and easy
interface for "going back in time" to figure out what happened.

I'd suggest the following:

1. soft deleting and instance does nothing with instance actions.

2. archiving instance (soft delete -> actually deleted) also archives
off instance actions.


I think this is also the right approach. Then we don't need to worry 
about adding soft delete for instance_actions, they are just archived 
when you archive the instances. It probably makes the logic in the 
archive code messier for this separate path, but it's looking like we're 
going to have to account for the bw_usage_cache table too (which has a 
uuid column for an instance but no foreign key back to the instances 
table and is not soft deleted).




3. update instance_actions API so that you can get instance_actions for
deleted instances (which I think doesn't work today).


Right, it doesn't. I was going to propose a spec for that since it's a 
simple API change with a microversion.




-Sean



--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Matt Riedemann]

On 11/20/2015 10:04 AM, Andrew Laski wrote:

On 11/20/15 at 09:51am, Matt Riedemann wrote:



On 11/20/2015 8:18 AM, Sean Dague wrote:

On 11/17/2015 10:51 PM, Matt Riedemann wrote:



I *don't* see any DB APIs for deleting instance actions.

Kind of an important difference there.  Jay got it at least. :)



Were we just planning on instance_actions living forever in the
database?

Should we soft delete instance_actions when we delete the referenced
instance?

Or should we (hard) delete instance_actions when we archive (move to
shadow tables) soft deleted instances?

This is going to be a blocker to getting nova-manage db
archive_deleted_rows working.

[1] https://review.openstack.org/#/c/246635/


instance_actions seems extremely useful, and at the ops meetups I've
been to has been one of the favorite features because it allows and easy
interface for "going back in time" to figure out what happened.

I'd suggest the following:

1. soft deleting and instance does nothing with instance actions.

2. archiving instance (soft delete -> actually deleted) also archives
off instance actions.


I think this is also the right approach. Then we don't need to worry
about adding soft delete for instance_actions, they are just archived
when you archive the instances. It probably makes the logic in the
archive code messier for this separate path, but it's looking like
we're going to have to account for the bw_usage_cache table too (which
has a uuid column for an instance but no foreign key back to the
instances table and is not soft deleted).



3. update instance_actions API so that you can get instance_actions for
deleted instances (which I think doesn't work today).


Right, it doesn't. I was going to propose a spec for that since it's a
simple API change with a microversion.


Adding a simple flag to expose instance actions for a deleted instance
if you know the uuid of the deleted instance will provide some
usefulness.  It does lack the discoverability of knowing that you had
*some* instance that was deleted and you don't have the uuid but want to
get at the deleted actions.  I would like to avoid bolting that onto
instance actions and keep that as a use case for an eventual Task API.





-Sean



--

Thanks,

Matt Riedemann


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



If you're an admin, you can list deleted instances using:

nova list --deleted

Or could, if we weren't busted on that right now [1].

So the use case I'm thinking of here is:

1. Multiple users are in the same project/tenant.
2. User A deletes an instance.
3. User B is wondering where the instance went, so they open a support 
ticket.
4. The admin checks for deleted instances on that project, finds the one 
in question.
5. Calls off to os-instance-actions with that instance uuid to see the 
deleted action and the user that did it (user A).

6. Closes the ticket saying that user A deleted the instance.
7. User B punches user A in the gut.

[1] https://bugs.launchpad.net/nova/+bug/1518382

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Andrew Laski]

On 11/20/15 at 09:51am, Matt Riedemann wrote:



On 11/20/2015 8:18 AM, Sean Dague wrote:

On 11/17/2015 10:51 PM, Matt Riedemann wrote:



I *don't* see any DB APIs for deleting instance actions.

Kind of an important difference there.  Jay got it at least. :)



Were we just planning on instance_actions living forever in the database?

Should we soft delete instance_actions when we delete the referenced
instance?

Or should we (hard) delete instance_actions when we archive (move to
shadow tables) soft deleted instances?

This is going to be a blocker to getting nova-manage db
archive_deleted_rows working.

[1] https://review.openstack.org/#/c/246635/


instance_actions seems extremely useful, and at the ops meetups I've
been to has been one of the favorite features because it allows and easy
interface for "going back in time" to figure out what happened.

I'd suggest the following:

1. soft deleting and instance does nothing with instance actions.

2. archiving instance (soft delete -> actually deleted) also archives
off instance actions.


I think this is also the right approach. Then we don't need to worry 
about adding soft delete for instance_actions, they are just archived 
when you archive the instances. It probably makes the logic in the 
archive code messier for this separate path, but it's looking like 
we're going to have to account for the bw_usage_cache table too 
(which has a uuid column for an instance but no foreign key back to 
the instances table and is not soft deleted).




3. update instance_actions API so that you can get instance_actions for
deleted instances (which I think doesn't work today).


Right, it doesn't. I was going to propose a spec for that since it's 
a simple API change with a microversion.


Adding a simple flag to expose instance actions for a deleted instance 
if you know the uuid of the deleted instance will provide some 
usefulness.  It does lack the discoverability of knowing that you had 
*some* instance that was deleted and you don't have the uuid but want to 
get at the deleted actions.  I would like to avoid bolting that onto 
instance actions and keep that as a use case for an eventual Task API.






-Sean



--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Sean Dague]

On 11/20/2015 11:36 AM, Matt Riedemann wrote:
> 
> 
> On 11/20/2015 10:04 AM, Andrew Laski wrote:
>> On 11/20/15 at 09:51am, Matt Riedemann wrote:
>>>
>>>
>>> On 11/20/2015 8:18 AM, Sean Dague wrote:
 On 11/17/2015 10:51 PM, Matt Riedemann wrote:
 
>
> I *don't* see any DB APIs for deleting instance actions.
>
> Kind of an important difference there.  Jay got it at least. :)
>
>>
>> Were we just planning on instance_actions living forever in the
>> database?
>>
>> Should we soft delete instance_actions when we delete the referenced
>> instance?
>>
>> Or should we (hard) delete instance_actions when we archive (move to
>> shadow tables) soft deleted instances?
>>
>> This is going to be a blocker to getting nova-manage db
>> archive_deleted_rows working.
>>
>> [1] https://review.openstack.org/#/c/246635/

 instance_actions seems extremely useful, and at the ops meetups I've
 been to has been one of the favorite features because it allows and
 easy
 interface for "going back in time" to figure out what happened.

 I'd suggest the following:

 1. soft deleting and instance does nothing with instance actions.

 2. archiving instance (soft delete -> actually deleted) also archives
 off instance actions.
>>>
>>> I think this is also the right approach. Then we don't need to worry
>>> about adding soft delete for instance_actions, they are just archived
>>> when you archive the instances. It probably makes the logic in the
>>> archive code messier for this separate path, but it's looking like
>>> we're going to have to account for the bw_usage_cache table too (which
>>> has a uuid column for an instance but no foreign key back to the
>>> instances table and is not soft deleted).
>>>

 3. update instance_actions API so that you can get instance_actions for
 deleted instances (which I think doesn't work today).
>>>
>>> Right, it doesn't. I was going to propose a spec for that since it's a
>>> simple API change with a microversion.
>>
>> Adding a simple flag to expose instance actions for a deleted instance
>> if you know the uuid of the deleted instance will provide some
>> usefulness.  It does lack the discoverability of knowing that you had
>> *some* instance that was deleted and you don't have the uuid but want to
>> get at the deleted actions.  I would like to avoid bolting that onto
>> instance actions and keep that as a use case for an eventual Task API.
>>
>>>

 -Sean

>>>
>>> -- 
>>>
>>> Thanks,
>>>
>>> Matt Riedemann
>>>
>>>
>>> __
>>>
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> If you're an admin, you can list deleted instances using:
> 
> nova list --deleted
> 
> Or could, if we weren't busted on that right now [1].
> 
> So the use case I'm thinking of here is:
> 
> 1. Multiple users are in the same project/tenant.
> 2. User A deletes an instance.
> 3. User B is wondering where the instance went, so they open a support
> ticket.
> 4. The admin checks for deleted instances on that project, finds the one
> in question.
> 5. Calls off to os-instance-actions with that instance uuid to see the
> deleted action and the user that did it (user A).
> 6. Closes the ticket saying that user A deleted the instance.
> 7. User B punches user A in the gut.
> 
> [1] https://bugs.launchpad.net/nova/+bug/1518382

+1

I think we need that on a T-shirt


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[melanie witt]

On Nov 20, 2015, at 6:18, Sean Dague  wrote:

> instance_actions seems extremely useful, and at the ops meetups I've
> been to has been one of the favorite features because it allows and easy
> interface for "going back in time" to figure out what happened.

Agreed, we're using it because it's such a quick and easy way to see what 
actions have been taken on an instance when users need support. We're not yet 
collecting notifications from the queue -- we do have them being dumped to the 
logs that are splunk searchable. So far, it hasn't been "easy" to look at 
instance action history that way.

> I'd suggest the following:
> 
> 1. soft deleting and instance does nothing with instance actions.
> 
> 2. archiving instance (soft delete -> actually deleted) also archives
> off instance actions.
> 
> 3. update instance_actions API so that you can get instance_actions for
> deleted instances (which I think doesn't work today).

+1

I kept trying to craft a reply to this thread and fortunately I waited long 
enough that someone else said exactly what I was trying to say.

-melanie (irc: melwitt)







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Matt Riedemann]

On 11/20/2015 3:00 PM, Sylvain Bauza wrote:



Le 20/11/2015 17:36, Matt Riedemann a écrit :



On 11/20/2015 10:04 AM, Andrew Laski wrote:

On 11/20/15 at 09:51am, Matt Riedemann wrote:



On 11/20/2015 8:18 AM, Sean Dague wrote:

On 11/17/2015 10:51 PM, Matt Riedemann wrote:



I *don't* see any DB APIs for deleting instance actions.

Kind of an important difference there.  Jay got it at least. :)



Were we just planning on instance_actions living forever in the
database?

Should we soft delete instance_actions when we delete the referenced
instance?

Or should we (hard) delete instance_actions when we archive (move to
shadow tables) soft deleted instances?

This is going to be a blocker to getting nova-manage db
archive_deleted_rows working.

[1] https://review.openstack.org/#/c/246635/


instance_actions seems extremely useful, and at the ops meetups I've
been to has been one of the favorite features because it allows and
easy
interface for "going back in time" to figure out what happened.

I'd suggest the following:

1. soft deleting and instance does nothing with instance actions.

2. archiving instance (soft delete -> actually deleted) also archives
off instance actions.


I think this is also the right approach. Then we don't need to worry
about adding soft delete for instance_actions, they are just archived
when you archive the instances. It probably makes the logic in the
archive code messier for this separate path, but it's looking like
we're going to have to account for the bw_usage_cache table too (which
has a uuid column for an instance but no foreign key back to the
instances table and is not soft deleted).



3. update instance_actions API so that you can get instance_actions
for
deleted instances (which I think doesn't work today).


Right, it doesn't. I was going to propose a spec for that since it's a
simple API change with a microversion.


Adding a simple flag to expose instance actions for a deleted instance
if you know the uuid of the deleted instance will provide some
usefulness.  It does lack the discoverability of knowing that you had
*some* instance that was deleted and you don't have the uuid but want to
get at the deleted actions.  I would like to avoid bolting that onto
instance actions and keep that as a use case for an eventual Task API.





-Sean



--

Thanks,

Matt Riedemann


__


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



If you're an admin, you can list deleted instances using:

nova list --deleted

Or could, if we weren't busted on that right now [1].

So the use case I'm thinking of here is:

1. Multiple users are in the same project/tenant.
2. User A deletes an instance.
3. User B is wondering where the instance went, so they open a support
ticket.
4. The admin checks for deleted instances on that project, finds the
one in question.
5. Calls off to os-instance-actions with that instance uuid to see the
deleted action and the user that did it (user A).
6. Closes the ticket saying that user A deleted the instance.
7. User B punches user A in the gut.

[1] https://bugs.launchpad.net/nova/+bug/1518382



Okay, that seems a good usecase for operators. Coolness, I'm fine with
soft-deleting instance_actions and provide a microversion for getting
actions for a known instance UUID, like Andrew said.


The plan right now (at least agreed to between myself and sdague) is not 
to soft delete instance actions, but to archive and hard-delete them 
when archiving instances.


As for allowing lookups on instance_actions for deleted instances, I 
plan on working that via this blueprint (still need to write the spec):


https://blueprints.launchpad.net/nova/+spec/os-instance-actions-read-deleted-instances





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Sylvain Bauza]

Le 20/11/2015 17:36, Matt Riedemann a écrit :



On 11/20/2015 10:04 AM, Andrew Laski wrote:

On 11/20/15 at 09:51am, Matt Riedemann wrote:



On 11/20/2015 8:18 AM, Sean Dague wrote:

On 11/17/2015 10:51 PM, Matt Riedemann wrote:



I *don't* see any DB APIs for deleting instance actions.

Kind of an important difference there.  Jay got it at least. :)



Were we just planning on instance_actions living forever in the
database?

Should we soft delete instance_actions when we delete the referenced
instance?

Or should we (hard) delete instance_actions when we archive (move to
shadow tables) soft deleted instances?

This is going to be a blocker to getting nova-manage db
archive_deleted_rows working.

[1] https://review.openstack.org/#/c/246635/


instance_actions seems extremely useful, and at the ops meetups I've
been to has been one of the favorite features because it allows and 
easy

interface for "going back in time" to figure out what happened.

I'd suggest the following:

1. soft deleting and instance does nothing with instance actions.

2. archiving instance (soft delete -> actually deleted) also archives
off instance actions.


I think this is also the right approach. Then we don't need to worry
about adding soft delete for instance_actions, they are just archived
when you archive the instances. It probably makes the logic in the
archive code messier for this separate path, but it's looking like
we're going to have to account for the bw_usage_cache table too (which
has a uuid column for an instance but no foreign key back to the
instances table and is not soft deleted).



3. update instance_actions API so that you can get instance_actions 
for

deleted instances (which I think doesn't work today).


Right, it doesn't. I was going to propose a spec for that since it's a
simple API change with a microversion.


Adding a simple flag to expose instance actions for a deleted instance
if you know the uuid of the deleted instance will provide some
usefulness.  It does lack the discoverability of knowing that you had
*some* instance that was deleted and you don't have the uuid but want to
get at the deleted actions.  I would like to avoid bolting that onto
instance actions and keep that as a use case for an eventual Task API.





-Sean



--

Thanks,

Matt Riedemann


__ 



OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



If you're an admin, you can list deleted instances using:

nova list --deleted

Or could, if we weren't busted on that right now [1].

So the use case I'm thinking of here is:

1. Multiple users are in the same project/tenant.
2. User A deletes an instance.
3. User B is wondering where the instance went, so they open a support 
ticket.
4. The admin checks for deleted instances on that project, finds the 
one in question.
5. Calls off to os-instance-actions with that instance uuid to see the 
deleted action and the user that did it (user A).

6. Closes the ticket saying that user A deleted the instance.
7. User B punches user A in the gut.

[1] https://bugs.launchpad.net/nova/+bug/1518382



Okay, that seems a good usecase for operators. Coolness, I'm fine with 
soft-deleting instance_actions and provide a microversion for getting 
actions for a known instance UUID, like Andrew said.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Andrew Laski]

On 11/18/15 at 08:32am, Andrew Laski wrote:

On 11/17/15 at 07:43pm, Jay Pipes wrote:

On 11/17/2015 05:43 PM, Matt Riedemann wrote:

I found some time to work on a reverse sort of nova's tables for the db
archive command, that looks like [1].  It works fine in the unit tests,
but fails because the deleted instances are referenced by
instance_actions that aren't deleted.  I see any DB APIs for deleting
instance actions.

Were we just planning on instance_actions living forever in the database?


Not as far as I understand.


They were never intended to live forever.  However there is a use 
case for holding on to the deleted action so that someone could query 
when or by whom their instance was deleted.  But the current API does 
not provide a good way to query for that so this may be something 
better left to the growing list of things that Tasks could address.





Should we soft delete instance_actions when we delete the referenced
instance?


No.


A few of us discussed this in #openstack-nova and highlighted that soft 
deleting them would make it easier to find things to purge/archive but 
would require handling older instance actions that were not soft 
deleted.  So there's a slight advantage to them after a data cleanup of 
old ones, but no strong technical advantage to going either way here.





Or should we (hard) delete instance_actions when we archive (move to
shadow tables) soft deleted instances?


Yes.


This seems good to me.  Though we probably want operator feedback on 
whether it's important to them to have these archived as well.




Best,
-jay


This is going to be a blocker to getting nova-manage db
archive_deleted_rows working.

[1] https://review.openstack.org/#/c/246635/



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Matt Riedemann]

On 11/18/2015 7:32 AM, Andrew Laski wrote:

On 11/17/15 at 07:43pm, Jay Pipes wrote:

On 11/17/2015 05:43 PM, Matt Riedemann wrote:

I found some time to work on a reverse sort of nova's tables for the db
archive command, that looks like [1].  It works fine in the unit tests,
but fails because the deleted instances are referenced by
instance_actions that aren't deleted.  I see any DB APIs for deleting
instance actions.

Were we just planning on instance_actions living forever in the
database?


Not as far as I understand.


They were never intended to live forever.  However there is a use case
for holding on to the deleted action so that someone could query when or
by whom their instance was deleted.  But the current API does not
provide a good way to query for that so this may be something better
left to the growing list of things that Tasks could address.




Should we soft delete instance_actions when we delete the referenced
instance?


No.


Or should we (hard) delete instance_actions when we archive (move to
shadow tables) soft deleted instances?


Yes.

Best,
-jay


This is going to be a blocker to getting nova-manage db
archive_deleted_rows working.

[1] https://review.openstack.org/#/c/246635/



__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



OK, so we talked a bit about this in IRC today.  I'll try to summarize 
here.  There are really two options it sounds like, either soft delete 
instance_actions when we soft delete instances, or just hard delete the 
instance_actions when archiving/purging soft deleted instances (to avoid 
the referential constraint failure when hard deleting the instances).


We have to note that the os-instance-actions API only works on instances 
that are not deleted [1]. So once you delete an instance (soft deleted 
in this case, i.e. instances.deleted != 0 in the DB), then the 
os-instance-actions API is useless. We could change that with a 
microversion to read deleted instances...but I'm not sure if anyone 
wants that (operators might?).


Even if we start soft deleting instance_actions, the archive/purge code 
will still have to account for existing resources in the database and 
handle them, i.e. when archiving an instance, find it's related 
instance_actions and if they are not soft deleted, soft delete and 
archive them first before archiving the instance. We could probably also 
provide a DB migration to do this, but that would be a lot of queries 
and updates potentially during an offline upgrade. We could also provide 
a nova-manage command to do it explicitly.


So hard-deleting instance_actions when archiving/purging instances is 
easier from a development perspective, it's much more straight forward 
(fewer bugs, yay!).  The downside to hard delete is they are gone, they 
wouldn't be in shadow tables or anything in the nova database.


The proposed remedy for wanting to store instance_actions after 
hard-delete is to basically have a monitoring service setup that is 
storing the instance delete notificiations, i.e. StackTach, gnocchi, 
probably others (monasca?). This is kind of a cop out for the nova dev 
team since it puts the burden on the operators to have this extra 
service running, but I suspect most already have something like this 
running anyway.


I'll cross post this to the operators ML and have it on this weeks' nova 
meeting agenda to see if we can find some agreement on a path forward here.


[1] 
https://github.com/openstack/nova/blob/12.0.0/nova/api/openstack/compute/instance_actions.py#L68


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Andrew Laski]

On 11/17/15 at 07:43pm, Jay Pipes wrote:

On 11/17/2015 05:43 PM, Matt Riedemann wrote:

I found some time to work on a reverse sort of nova's tables for the db
archive command, that looks like [1].  It works fine in the unit tests,
but fails because the deleted instances are referenced by
instance_actions that aren't deleted.  I see any DB APIs for deleting
instance actions.

Were we just planning on instance_actions living forever in the database?


Not as far as I understand.


They were never intended to live forever.  However there is a use case 
for holding on to the deleted action so that someone could query when or 
by whom their instance was deleted.  But the current API does not 
provide a good way to query for that so this may be something better 
left to the growing list of things that Tasks could address.





Should we soft delete instance_actions when we delete the referenced
instance?


No.


Or should we (hard) delete instance_actions when we archive (move to
shadow tables) soft deleted instances?


Yes.

Best,
-jay


This is going to be a blocker to getting nova-manage db
archive_deleted_rows working.

[1] https://review.openstack.org/#/c/246635/



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Matt Riedemann]

On 10/9/2015 1:16 PM, Matt Riedemann wrote:



On 10/9/2015 12:03 PM, Jay Pipes wrote:

On 10/07/2015 11:04 AM, Matt Riedemann wrote:

I'm wondering why we don't reverse sort the tables using the sqlalchemy
metadata object before processing the tables for delete?  That's the
same thing I did in the 267 migration since we needed to process the
tree starting with the leafs and then eventually get back to the
instances table (since most roads lead to the instances table).


Yes, that would make a lot of sense to me if we used the SA metadata
object for reverse sorting.


When I get some free time next week I'm going to play with this.




Another thing that's really weird is how max_rows is used in this code.
There is cumulative tracking of the max_rows value so if the value you
pass in is too small, you might not actually be removing anything.

I figured max_rows meant up to max_rows from each table, not max_rows
*total* across all tables. By my count, there are 52 tables in the nova
db model. The way I read the code, if I pass in max_rows=10 and say it
processes table A and archives 7 rows, then when it processes table B it
will pass max_rows=(max_rows - rows_archived), which would be 3 for
table B. If we archive 3 rows from table B, rows_archived >= max_rows
and we quit. So to really make this work, you have to pass in something
big for max_rows, like 1000, which seems completely random.

Does this seem odd to anyone else?


Uhm, yes it does.

 > Given the relationships between

tables, I'd think you'd want to try and delete max_rows for all tables,
so archive 10 instances, 10 block_device_mapping, 10 pci_devices, etc.

I'm also bringing this up now because there is a thread in the operators
list which pointed me to a set of scripts that operators at GoDaddy are
using for archiving deleted rows:

http://lists.openstack.org/pipermail/openstack-operators/2015-October/008392.html



Presumably because the command in nova doesn't work. We should either
make this thing work or just punt and delete it because no one cares.


The db archive code in Nova just doesn't make much sense to me at all.
The algorithm for purging stuff, like you mention above, does not take
into account the relationships between tables; instead of diving into
the children relations and archiving those first, the code just uses a
simplistic "well, if we hit a foreign key error, just ignore and
continue archiving other things, we will eventually repeat the call to
delete this row" strategy:

https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L6021-L6023



Yeah, I noticed that too and I don't think it actually does anything. We
never actually come back since that would require some
tracking/stack/recursion stuff to retry failed tables, which we don't do.




I had a proposal [1] to completely rework the whole shadow table mess
and db archiving functionality. I continue to believe that is the
appropriate solution for this, and that we should rip out the existing
functionality because it simply does not work properly.

Best,
-jay

[1] https://review.openstack.org/#/c/137669/


Are you going to pick that back up? Or sick some minions on it.



__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





I found some time to work on a reverse sort of nova's tables for the db 
archive command, that looks like [1].  It works fine in the unit tests, 
but fails because the deleted instances are referenced by 
instance_actions that aren't deleted.  I see any DB APIs for deleting 
instance actions.


Were we just planning on instance_actions living forever in the database?

Should we soft delete instance_actions when we delete the referenced 
instance?


Or should we (hard) delete instance_actions when we archive (move to 
shadow tables) soft deleted instances?


This is going to be a blocker to getting nova-manage db 
archive_deleted_rows working.


[1] https://review.openstack.org/#/c/246635/

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Jay Pipes]

On 11/17/2015 05:43 PM, Matt Riedemann wrote:

I found some time to work on a reverse sort of nova's tables for the db
archive command, that looks like [1].  It works fine in the unit tests,
but fails because the deleted instances are referenced by
instance_actions that aren't deleted.  I see any DB APIs for deleting
instance actions.

Were we just planning on instance_actions living forever in the database?


Not as far as I understand.


Should we soft delete instance_actions when we delete the referenced
instance?


No.


Or should we (hard) delete instance_actions when we archive (move to
shadow tables) soft deleted instances?


Yes.

Best,
-jay


This is going to be a blocker to getting nova-manage db
archive_deleted_rows working.

[1] https://review.openstack.org/#/c/246635/



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Matt Riedemann]

On 11/17/2015 7:43 PM, Matt Riedemann wrote:



On 10/9/2015 1:16 PM, Matt Riedemann wrote:



On 10/9/2015 12:03 PM, Jay Pipes wrote:

On 10/07/2015 11:04 AM, Matt Riedemann wrote:

I'm wondering why we don't reverse sort the tables using the sqlalchemy
metadata object before processing the tables for delete?  That's the
same thing I did in the 267 migration since we needed to process the
tree starting with the leafs and then eventually get back to the
instances table (since most roads lead to the instances table).


Yes, that would make a lot of sense to me if we used the SA metadata
object for reverse sorting.


When I get some free time next week I'm going to play with this.




Another thing that's really weird is how max_rows is used in this code.
There is cumulative tracking of the max_rows value so if the value you
pass in is too small, you might not actually be removing anything.

I figured max_rows meant up to max_rows from each table, not max_rows
*total* across all tables. By my count, there are 52 tables in the nova
db model. The way I read the code, if I pass in max_rows=10 and say it
processes table A and archives 7 rows, then when it processes table
B it
will pass max_rows=(max_rows - rows_archived), which would be 3 for
table B. If we archive 3 rows from table B, rows_archived >= max_rows
and we quit. So to really make this work, you have to pass in something
big for max_rows, like 1000, which seems completely random.

Does this seem odd to anyone else?


Uhm, yes it does.

 > Given the relationships between

tables, I'd think you'd want to try and delete max_rows for all tables,
so archive 10 instances, 10 block_device_mapping, 10 pci_devices, etc.

I'm also bringing this up now because there is a thread in the
operators
list which pointed me to a set of scripts that operators at GoDaddy are
using for archiving deleted rows:

http://lists.openstack.org/pipermail/openstack-operators/2015-October/008392.html




Presumably because the command in nova doesn't work. We should either
make this thing work or just punt and delete it because no one cares.


The db archive code in Nova just doesn't make much sense to me at all.
The algorithm for purging stuff, like you mention above, does not take
into account the relationships between tables; instead of diving into
the children relations and archiving those first, the code just uses a
simplistic "well, if we hit a foreign key error, just ignore and
continue archiving other things, we will eventually repeat the call to
delete this row" strategy:

https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L6021-L6023




Yeah, I noticed that too and I don't think it actually does anything. We
never actually come back since that would require some
tracking/stack/recursion stuff to retry failed tables, which we don't do.




I had a proposal [1] to completely rework the whole shadow table mess
and db archiving functionality. I continue to believe that is the
appropriate solution for this, and that we should rip out the existing
functionality because it simply does not work properly.

Best,
-jay

[1] https://review.openstack.org/#/c/137669/


Are you going to pick that back up? Or sick some minions on it.



__


OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





I found some time to work on a reverse sort of nova's tables for the db
archive command, that looks like [1].  It works fine in the unit tests,
but fails because the deleted instances are referenced by
instance_actions that aren't deleted.  I see any DB APIs for deleting
instance actions.


I *don't* see any DB APIs for deleting instance actions.

Kind of an important difference there.  Jay got it at least. :)



Were we just planning on instance_actions living forever in the database?

Should we soft delete instance_actions when we delete the referenced
instance?

Or should we (hard) delete instance_actions when we archive (move to
shadow tables) soft deleted instances?

This is going to be a blocker to getting nova-manage db
archive_deleted_rows working.

[1] https://review.openstack.org/#/c/246635/



--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Jay Pipes]

On 10/07/2015 11:04 AM, Matt Riedemann wrote:

I'm wondering why we don't reverse sort the tables using the sqlalchemy
metadata object before processing the tables for delete?  That's the
same thing I did in the 267 migration since we needed to process the
tree starting with the leafs and then eventually get back to the
instances table (since most roads lead to the instances table).


Yes, that would make a lot of sense to me if we used the SA metadata 
object for reverse sorting.



Another thing that's really weird is how max_rows is used in this code.
There is cumulative tracking of the max_rows value so if the value you
pass in is too small, you might not actually be removing anything.

I figured max_rows meant up to max_rows from each table, not max_rows
*total* across all tables. By my count, there are 52 tables in the nova
db model. The way I read the code, if I pass in max_rows=10 and say it
processes table A and archives 7 rows, then when it processes table B it
will pass max_rows=(max_rows - rows_archived), which would be 3 for
table B. If we archive 3 rows from table B, rows_archived >= max_rows
and we quit. So to really make this work, you have to pass in something
big for max_rows, like 1000, which seems completely random.

Does this seem odd to anyone else?


Uhm, yes it does.

> Given the relationships between

tables, I'd think you'd want to try and delete max_rows for all tables,
so archive 10 instances, 10 block_device_mapping, 10 pci_devices, etc.

I'm also bringing this up now because there is a thread in the operators
list which pointed me to a set of scripts that operators at GoDaddy are
using for archiving deleted rows:

http://lists.openstack.org/pipermail/openstack-operators/2015-October/008392.html

Presumably because the command in nova doesn't work. We should either
make this thing work or just punt and delete it because no one cares.


The db archive code in Nova just doesn't make much sense to me at all. 
The algorithm for purging stuff, like you mention above, does not take 
into account the relationships between tables; instead of diving into 
the children relations and archiving those first, the code just uses a 
simplistic "well, if we hit a foreign key error, just ignore and 
continue archiving other things, we will eventually repeat the call to 
delete this row" strategy:


https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L6021-L6023

I had a proposal [1] to completely rework the whole shadow table mess 
and db archiving functionality. I continue to believe that is the 
appropriate solution for this, and that we should rip out the existing 
functionality because it simply does not work properly.


Best,
-jay

[1] https://review.openstack.org/#/c/137669/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Matt Riedemann]

On 10/9/2015 12:03 PM, Jay Pipes wrote:

On 10/07/2015 11:04 AM, Matt Riedemann wrote:

I'm wondering why we don't reverse sort the tables using the sqlalchemy
metadata object before processing the tables for delete?  That's the
same thing I did in the 267 migration since we needed to process the
tree starting with the leafs and then eventually get back to the
instances table (since most roads lead to the instances table).


Yes, that would make a lot of sense to me if we used the SA metadata
object for reverse sorting.


When I get some free time next week I'm going to play with this.




Another thing that's really weird is how max_rows is used in this code.
There is cumulative tracking of the max_rows value so if the value you
pass in is too small, you might not actually be removing anything.

I figured max_rows meant up to max_rows from each table, not max_rows
*total* across all tables. By my count, there are 52 tables in the nova
db model. The way I read the code, if I pass in max_rows=10 and say it
processes table A and archives 7 rows, then when it processes table B it
will pass max_rows=(max_rows - rows_archived), which would be 3 for
table B. If we archive 3 rows from table B, rows_archived >= max_rows
and we quit. So to really make this work, you have to pass in something
big for max_rows, like 1000, which seems completely random.

Does this seem odd to anyone else?


Uhm, yes it does.

 > Given the relationships between

tables, I'd think you'd want to try and delete max_rows for all tables,
so archive 10 instances, 10 block_device_mapping, 10 pci_devices, etc.

I'm also bringing this up now because there is a thread in the operators
list which pointed me to a set of scripts that operators at GoDaddy are
using for archiving deleted rows:

http://lists.openstack.org/pipermail/openstack-operators/2015-October/008392.html


Presumably because the command in nova doesn't work. We should either
make this thing work or just punt and delete it because no one cares.


The db archive code in Nova just doesn't make much sense to me at all.
The algorithm for purging stuff, like you mention above, does not take
into account the relationships between tables; instead of diving into
the children relations and archiving those first, the code just uses a
simplistic "well, if we hit a foreign key error, just ignore and
continue archiving other things, we will eventually repeat the call to
delete this row" strategy:

https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L6021-L6023


Yeah, I noticed that too and I don't think it actually does anything. We 
never actually come back since that would require some 
tracking/stack/recursion stuff to retry failed tables, which we don't do.





I had a proposal [1] to completely rework the whole shadow table mess
and db archiving functionality. I continue to believe that is the
appropriate solution for this, and that we should rip out the existing
functionality because it simply does not work properly.

Best,
-jay

[1] https://review.openstack.org/#/c/137669/


Are you going to pick that back up? Or sick some minions on it.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Jay Pipes]

On 10/09/2015 02:16 PM, Matt Riedemann wrote:

On 10/9/2015 12:03 PM, Jay Pipes wrote:

I had a proposal [1] to completely rework the whole shadow table mess
and db archiving functionality. I continue to believe that is the
appropriate solution for this, and that we should rip out the existing
functionality because it simply does not work properly.

Best,
-jay

[1] https://review.openstack.org/#/c/137669/


Are you going to pick that back up? Or sick some minions on it.


I don't personally have the bandwidth to do this. If anyone out there in 
Nova contributor land has interest, just find me on IRC. :)


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Matt Riedemann]

On 12/12/2014 7:54 PM, melanie witt wrote:

Hi everybody,

At some point, our db archiving functionality got broken because there was a 
change to stop ever deleting instance system metadata [1]. For those 
unfamiliar, the 'nova-manage db archive_deleted_rows' is the thing that moves 
all soft-deleted (deleted=nonzero) rows to the shadow tables. This is a 
periodic cleaning that operators can do to maintain performance (as things can 
get sluggish when deleted=nonzero rows accumulate).

The change was made because instance_type data still needed to be read even after 
instances had been deleted, because we allow admin to view deleted instances. I saw a bug 
[2] and two patches [3][4] which aimed to fix this by changing back to soft-deleting 
instance sysmeta when instances are deleted, and instead allow 
read_deleted="yes" for the things that need to read instance_type for deleted 
instances present in the db.

My question is, is this approach okay? If so, I'd like to see these patches 
revive so we can have our db archiving working again. :) I think there's likely 
something I'm missing about the approach, so I'm hoping people who know more 
about instance sysmeta than I do, can chime in on how/if we can fix this for db 
archiving. Thanks.

[1] https://bugs.launchpad.net/nova/+bug/1185190
[2] https://bugs.launchpad.net/nova/+bug/1226049
[3] https://review.openstack.org/#/c/110875/
[4] https://review.openstack.org/#/c/109201/

melanie (melwitt)






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



I'd like to bring this back up since even though [3] and [4] are merged, 
nova-manage db archive_deleted_rows still fails to delete rows from some 
tables because of foreign key constraint issues, detailed here:


https://bugs.launchpad.net/nova/+bug/1183523/comments/12

I'm wondering why we don't reverse sort the tables using the sqlalchemy 
metadata object before processing the tables for delete?  That's the 
same thing I did in the 267 migration since we needed to process the 
tree starting with the leafs and then eventually get back to the 
instances table (since most roads lead to the instances table).


Another thing that's really weird is how max_rows is used in this code. 
There is cumulative tracking of the max_rows value so if the value you 
pass in is too small, you might not actually be removing anything.


I figured max_rows meant up to max_rows from each table, not max_rows 
*total* across all tables. By my count, there are 52 tables in the nova 
db model. The way I read the code, if I pass in max_rows=10 and say it 
processes table A and archives 7 rows, then when it processes table B it 
will pass max_rows=(max_rows - rows_archived), which would be 3 for 
table B. If we archive 3 rows from table B, rows_archived >= max_rows 
and we quit. So to really make this work, you have to pass in something 
big for max_rows, like 1000, which seems completely random.


Does this seem odd to anyone else?  Given the relationships between 
tables, I'd think you'd want to try and delete max_rows for all tables, 
so archive 10 instances, 10 block_device_mapping, 10 pci_devices, etc.


I'm also bringing this up now because there is a thread in the operators 
list which pointed me to a set of scripts that operators at GoDaddy are 
using for archiving deleted rows:


http://lists.openstack.org/pipermail/openstack-operators/2015-October/008392.html

Presumably because the command in nova doesn't work. We should either 
make this thing work or just punt and delete it because no one cares.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Matt Riedemann]

On 12/12/2014 7:54 PM, melanie witt wrote:

Hi everybody,

At some point, our db archiving functionality got broken because there was a 
change to stop ever deleting instance system metadata [1]. For those 
unfamiliar, the 'nova-manage db archive_deleted_rows' is the thing that moves 
all soft-deleted (deleted=nonzero) rows to the shadow tables. This is a 
periodic cleaning that operators can do to maintain performance (as things can 
get sluggish when deleted=nonzero rows accumulate).

The change was made because instance_type data still needed to be read even after 
instances had been deleted, because we allow admin to view deleted instances. I saw a bug 
[2] and two patches [3][4] which aimed to fix this by changing back to soft-deleting 
instance sysmeta when instances are deleted, and instead allow 
read_deleted=yes for the things that need to read instance_type for deleted 
instances present in the db.

My question is, is this approach okay? If so, I'd like to see these patches 
revive so we can have our db archiving working again. :) I think there's likely 
something I'm missing about the approach, so I'm hoping people who know more 
about instance sysmeta than I do, can chime in on how/if we can fix this for db 
archiving. Thanks.

[1] https://bugs.launchpad.net/nova/+bug/1185190
[2] https://bugs.launchpad.net/nova/+bug/1226049
[3] https://review.openstack.org/#/c/110875/
[4] https://review.openstack.org/#/c/109201/

melanie (melwitt)






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



I changed this from In Progress to Confirmed, removed Alex as the owner 
(since I didn't see any patches from him) and marked it High:


https://bugs.launchpad.net/nova/+bug/1226049

It looks like that could be a duplicate of bug 
https://bugs.launchpad.net/nova/+bug/1183523 which sounds like a lot of 
the same problems. dripton had looked at it at one point and said it was 
Won't Fix at that time, but I don't think that's the case.  Note comment 
7 in there:


https://bugs.launchpad.net/nova/+bug/1183523/comments/7

comstud thinks we can fix this but we need to do instance_type data 
differently. Maybe embedded JSON blobs so we have all the information we 
need without a reference to the instances row. (My opinion: yuck.) So 
this bug is staying open for now, but it requires some significant 
redesign to fix.


I'm not sure if that's related to comstud's instance_type design summit 
topic in Atlanta or not, it sounds the same:


http://junodesignsummit.sched.org/event/e3f1d51c53fc484d070f02ea36d08601#.VJCS6yvF-KU

I can't find the etherpad for that. I'm wondering if Dan Smith's 
blueprint for flavor-from-sysmeta-to-blob handles that? [1] I've never 
been sure how those two items are related.


Anyway, I think the fix is for the taking assuming someone has a good 
fix.  As noted in one of the bugs, the foreign key constraints in nova 
don't have cascading deletes, so if it's a foreign key issue, we should 
find the one that's not being cleaned up before the delete.  It looked 
like dripton thought it was fixed_ips at one point:


https://review.openstack.org/#/c/32742/

[1] 
http://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/flavor-from-sysmeta-to-blob.html


--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage db archive_deleted_rows broken

[Andrew Laski]

On 12/12/2014 08:54 PM, melanie witt wrote:

Hi everybody,

At some point, our db archiving functionality got broken because there was a 
change to stop ever deleting instance system metadata [1]. For those 
unfamiliar, the 'nova-manage db archive_deleted_rows' is the thing that moves 
all soft-deleted (deleted=nonzero) rows to the shadow tables. This is a 
periodic cleaning that operators can do to maintain performance (as things can 
get sluggish when deleted=nonzero rows accumulate).

The change was made because instance_type data still needed to be read even after 
instances had been deleted, because we allow admin to view deleted instances. I saw a bug 
[2] and two patches [3][4] which aimed to fix this by changing back to soft-deleting 
instance sysmeta when instances are deleted, and instead allow 
read_deleted=yes for the things that need to read instance_type for deleted 
instances present in the db.

My question is, is this approach okay? If so, I'd like to see these patches 
revive so we can have our db archiving working again. :) I think there's likely 
something I'm missing about the approach, so I'm hoping people who know more 
about instance sysmeta than I do, can chime in on how/if we can fix this for db 
archiving. Thanks.


I looked briefly into tackling this as well a while back.  The tricky 
piece that I hit is what system_metadata should be available when 
read_deleted='yes'.  Is it okay for it to be all deleted system_metadata 
or should it only be the system_metadata that was deleted at the same 
time as the instance?  I didn't get to dig in enough to answer that.


Also there are periodic tasks that query for deleted instances so those 
might need to pull system_metadata in addition to the API.





[1] https://bugs.launchpad.net/nova/+bug/1185190
[2] https://bugs.launchpad.net/nova/+bug/1226049
[3] https://review.openstack.org/#/c/110875/
[4] https://review.openstack.org/#/c/109201/

melanie (melwitt)






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev