Re: [Openstack-operators] How do you handle purge of database tables ?

[Matt Riedemann]

On 6/23/2016 10:21 AM, Belmiro Moreira wrote:

Hi,
we wrote this blog post a year ago but it still can be useful depending
on the
OpenStack version that you are running.

http://openstack-in-production.blogspot.ch/2015/05/purging-nova-databases-in-cell.html

Belmiro



This looks similar to what was being worked on in nova for the purge 
instances command [1]. As noted elsewhere in this thread, the blueprint 
for that is going to be re-proposed for Ocata since it missed the freeze 
for Newton.


It'd be good to collaborate on these tools within Nova so that we can 
unit test them with the rest of the code base to make sure we don't 
regress this functionality when new tables are added (we have a 
functional test that boots a server and then deletes it and makes sure 
we can archive/purge).


The main difference from the CERN script is going to be in the cells v1 
support, which is probably not something we'll bake into the command 
that lives in nova since cells v1 is frozen as we work on cells v2 support.


[1] https://review.openstack.org/#/c/203751/

--

Thanks,

Matt Riedemann


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[Matt Riedemann]

On 6/23/2016 8:32 AM, Nick Jones wrote:

On Thu, 2016-06-23 at 09:55 +0100, Nick Jones wrote:

Another vote of confidence for the script that Tim has mentioned with
regards to clearing down Nova’s DB.  I blogged a bit about the process
and the results here:


http://dischord.org/2015/12/30/archiving-data-in-nova-s-database/

—






In Nova, there's 'nova-manage db archive_deleted_rows' - is that the one
referred to in the blog post above that broke?


That’s the one.

—

-Nick



nova-manage db archive_deleted_rows is fixed in newton [1] which was 
also backported to stable/mitaka [2].


The purge-deleted-instances-cmd blueprint [3] wasn't re-proposed in time 
for Newton but will probably be done in Ocata as some new people have 
expressed interest in picking that back up.


Nova is also migrating some tables to the API database and as part of 
that we are dropping the SoftDeleteMixin so when things are deleted from 
the Nova API database they are gone, not soft deleted.


[1] https://review.openstack.org/#/c/299474/
[2] https://review.openstack.org/#/c/326730/
[3] https://blueprints.launchpad.net/nova/+spec/purge-deleted-instances-cmd

--

Thanks,

Matt Riedemann


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[Simon Leinen]

Nick Jones writes:
> Another vote of confidence for the script that Tim has mentioned with
> regards to clearing down Nova’s DB.  I blogged a bit about the process
> and the results here:
> http://dischord.org/2015/12/30/archiving-data-in-nova-s-database/

Unfortunately the nova/openstack_db_archive.sh seems to be broken by
https://review.openstack.org/#/c/269530/3 ; I have created a bug:

https://bugs.launchpad.net/osops/+bug/1596193

Can anyone confirm? I have a fix locally and will push a review soon.
-- 
Simon.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[Edgar Magana]

I would encourage you to use that repo and if there is something worth to be 
added updated, please do! All suggestions are welcome.

Edgar

From: Tim Bell <tim.b...@cern.ch>
Date: Wednesday, June 22, 2016 at 10:53 PM
To: Abel Lopez <alopg...@gmail.com>, Gilles Mocellin 
<gilles.mocel...@nuagelibre.org>
Cc: "OpenStack-operators@lists.openstack.org" 
<openstack-operators@lists.openstack.org>
Subject: Re: [Openstack-operators] How do you handle purge of database tables ?

There are also some tools in the OSOps repository (Nova for example has 
https://github.com/openstack/osops-tools-generic/tree/master/nova)<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_osops-2Dtools-2Dgeneric_tree_master_nova-29=CwMGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=YTsxdMdPQDeVWL18ZtfPm4l8Ukb65tAIMhgQ9NI5wtQ=iqH-TOGPCLIqE7no9W6uOVjn0xGpf-aV9YPh_whjx8Q=>

Tim

From: Abel Lopez <alopg...@gmail.com>
Date: Thursday 23 June 2016 at 00:03
To: Gilles Mocellin <gilles.mocel...@nuagelibre.org>
Cc: openstack-operators <openstack-operators@lists.openstack.org>
Subject: Re: [Openstack-operators] How do you handle purge of database tables ?

Some projects (e.g. cinder) have tools..
`cinder-manage db-purge` or maybe it's "db purge" or something along those 
lines...
It was around Kilo when that got added.

Otherwise, in the past, I've done
mysql backup, then a massive table by table purge of "deleted=1" rows.

On Wed, Jun 22, 2016 at 2:39 PM, Gilles Mocellin 
<gilles.mocel...@nuagelibre.org<mailto:gilles.mocel...@nuagelibre.org>> wrote:
Le 22/06/2016 à 23:26, Gilles Mocellin a écrit :
Hello,

While digging in nova's database, I found that many objects ar not really 
deleted, but instead just marked deleted.
In fact, it's a general behavior in other projects (cinder, glance...).

I understand that. It can be handy.

But, is there a way to handle regular purging of theses elements inside 
OpenStack configuration ?

Or do we, operators, have to create batchs to purge all that ?

I've found some blueprints indicating that there nothing for now, inside 
OpenStack :

https://blueprints.launchpad.net/nova/+spec/purge-deleted-instances-cmd<https://urldefense.proofpoint.com/v2/url?u=https-3A__blueprints.launchpad.net_nova_-26-2343-3Bspec_purge-2Ddeleted-2Dinstances-2Dcmd=CwMGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=YTsxdMdPQDeVWL18ZtfPm4l8Ukb65tAIMhgQ9NI5wtQ=NNbrSqzhDJ1ZxMs5D6pZKKrpsFj6PY1ssWnk6ITIQtA=>
https://blueprints.launchpad.net/cinder/+spec/database-purge<https://urldefense.proofpoint.com/v2/url?u=https-3A__blueprints.launchpad.net_cinder_-26-2343-3Bspec_database-2Dpurge=CwMGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=YTsxdMdPQDeVWL18ZtfPm4l8Ukb65tAIMhgQ9NI5wtQ=CeqEqcnBufMXJxtM88hq2RG0qxfrqP6Eq6WofpFhqeM=>
https://blueprints.launchpad.net/glance/+spec/database-purge<https://urldefense.proofpoint.com/v2/url?u=https-3A__blueprints.launchpad.net_glance_-26-2343-3Bspec_database-2Dpurge=CwMGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=YTsxdMdPQDeVWL18ZtfPm4l8Ukb65tAIMhgQ9NI5wtQ=iFf934niIjhaYHZLjcSD9-ebCVvpEeIOqljFy9unIfo=>



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Doperators=CwMGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=YTsxdMdPQDeVWL18ZtfPm4l8Ukb65tAIMhgQ9NI5wtQ=O4evdHuvokO4rzSqgxybzMqXAxxQJ88dRb4S3DJd-Xw=>

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[Belmiro Moreira]

Hi,
we wrote this blog post a year ago but it still can be useful depending on
the
OpenStack version that you are running.

http://openstack-in-production.blogspot.ch/2015/05/purging-nova-databases-in-cell.html

Belmiro

On Thu, Jun 23, 2016 at 3:32 PM, Nick Jones 
wrote:

> > On Thu, 2016-06-23 at 09:55 +0100, Nick Jones wrote:
> >> Another vote of confidence for the script that Tim has mentioned with
> >> regards to clearing down Nova’s DB.  I blogged a bit about the process
> >> and the results here:
> >>
> >>
> >> http://dischord.org/2015/12/30/archiving-data-in-nova-s-database/
> >>
> >> —
> >
> >>
> >
> > In Nova, there's 'nova-manage db archive_deleted_rows' - is that the one
> > referred to in the blog post above that broke?
>
> That’s the one.
>
> —
>
> -Nick
> --
> DataCentred Limited registered in England and Wales no. 05611763
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[Nick Jones]

> On Thu, 2016-06-23 at 09:55 +0100, Nick Jones wrote:
>> Another vote of confidence for the script that Tim has mentioned with
>> regards to clearing down Nova’s DB.  I blogged a bit about the process
>> and the results here:
>> 
>> 
>> http://dischord.org/2015/12/30/archiving-data-in-nova-s-database/
>> 
>> — 
> 
>> 
> 
> In Nova, there's 'nova-manage db archive_deleted_rows' - is that the one
> referred to in the blog post above that broke?

That’s the one.

— 

-Nick
-- 
DataCentred Limited registered in England and Wales no. 05611763

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[xav]

On Thu, 2016-06-23 at 09:55 +0100, Nick Jones wrote:
> Another vote of confidence for the script that Tim has mentioned with
> regards to clearing down Nova’s DB.  I blogged a bit about the process
> and the results here:
> 
> 
> http://dischord.org/2015/12/30/archiving-data-in-nova-s-database/
> 
> — 

> 

In Nova, there's 'nova-manage db archive_deleted_rows' - is that the one
referred to in the blog post above that broke?


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[Nick Jones]

Another vote of confidence for the script that Tim has mentioned with regards 
to clearing down Nova’s DB.  I blogged a bit about the process and the results 
here:

http://dischord.org/2015/12/30/archiving-data-in-nova-s-database/ 
<http://dischord.org/2015/12/30/archiving-data-in-nova-s-database/>

— 

-Nick

> On 23 Jun 2016, at 06:53, Tim Bell <tim.b...@cern.ch> wrote:
> 
> There are also some tools in the OSOps repository (Nova for example has 
> https://github.com/openstack/osops-tools-generic/tree/master/nova) 
> <https://github.com/openstack/osops-tools-generic/tree/master/nova)>
>  
> Tim
>  
> From: Abel Lopez <alopg...@gmail.com>
> Date: Thursday 23 June 2016 at 00:03
> To: Gilles Mocellin <gilles.mocel...@nuagelibre.org>
> Cc: openstack-operators <openstack-operators@lists.openstack.org>
> Subject: Re: [Openstack-operators] How do you handle purge of database tables 
> ?
>  
>> Some projects (e.g. cinder) have tools.. 
>> `cinder-manage db-purge` or maybe it's "db purge" or something along those 
>> lines...
>> It was around Kilo when that got added.
>>  
>> Otherwise, in the past, I've done
>> mysql backup, then a massive table by table purge of "deleted=1" rows.
>>  
>> On Wed, Jun 22, 2016 at 2:39 PM, Gilles Mocellin 
>> <gilles.mocel...@nuagelibre.org <mailto:gilles.mocel...@nuagelibre.org>> 
>> wrote:
>>> Le 22/06/2016 à 23:26, Gilles Mocellin a écrit :
>>>> Hello,
>>>> 
>>>> While digging in nova's database, I found that many objects ar not really 
>>>> deleted, but instead just marked deleted.
>>>> In fact, it's a general behavior in other projects (cinder, glance...).
>>>> 
>>>> I understand that. It can be handy.
>>>> 
>>>> But, is there a way to handle regular purging of theses elements inside 
>>>> OpenStack configuration ?
>>>> 
>>>> Or do we, operators, have to create batchs to purge all that ?
>>>> 
>>> 
>>> I've found some blueprints indicating that there nothing for now, inside 
>>> OpenStack :
>>> 
>>> https://blueprints.launchpad.net/nova/+spec/purge-deleted-instances-cmd 
>>> <https://blueprints.launchpad.net/nova/+spec/purge-deleted-instances-cmd>
>>> https://blueprints.launchpad.net/cinder/+spec/database-purge 
>>> <https://blueprints.launchpad.net/cinder/+spec/database-purge>
>>> https://blueprints.launchpad.net/glance/+spec/database-purge 
>>> <https://blueprints.launchpad.net/glance/+spec/database-purge>

-- 
DataCentred Limited registered in England and Wales no. 05611763
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[Tim Bell]

There are also some tools in the OSOps repository (Nova for example has 
https://github.com/openstack/osops-tools-generic/tree/master/nova)

Tim

From: Abel Lopez <alopg...@gmail.com>
Date: Thursday 23 June 2016 at 00:03
To: Gilles Mocellin <gilles.mocel...@nuagelibre.org>
Cc: openstack-operators <openstack-operators@lists.openstack.org>
Subject: Re: [Openstack-operators] How do you handle purge of database tables ?

Some projects (e.g. cinder) have tools..
`cinder-manage db-purge` or maybe it's "db purge" or something along those 
lines...
It was around Kilo when that got added.

Otherwise, in the past, I've done
mysql backup, then a massive table by table purge of "deleted=1" rows.

On Wed, Jun 22, 2016 at 2:39 PM, Gilles Mocellin 
<gilles.mocel...@nuagelibre.org<mailto:gilles.mocel...@nuagelibre.org>> wrote:
Le 22/06/2016 à 23:26, Gilles Mocellin a écrit :
Hello,

While digging in nova's database, I found that many objects ar not really 
deleted, but instead just marked deleted.
In fact, it's a general behavior in other projects (cinder, glance...).

I understand that. It can be handy.

But, is there a way to handle regular purging of theses elements inside 
OpenStack configuration ?

Or do we, operators, have to create batchs to purge all that ?

I've found some blueprints indicating that there nothing for now, inside 
OpenStack :

https://blueprints.launchpad.net/nova/+spec/purge-deleted-instances-cmd
https://blueprints.launchpad.net/cinder/+spec/database-purge
https://blueprints.launchpad.net/glance/+spec/database-purge



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[Abel Lopez]

Some projects (e.g. cinder) have tools..
`cinder-manage db-purge` or maybe it's "db purge" or something along those
lines...
It was around Kilo when that got added.

Otherwise, in the past, I've done
mysql backup, then a massive table by table purge of "deleted=1" rows.

On Wed, Jun 22, 2016 at 2:39 PM, Gilles Mocellin <
gilles.mocel...@nuagelibre.org> wrote:

> Le 22/06/2016 à 23:26, Gilles Mocellin a écrit :
>
>> Hello,
>>
>> While digging in nova's database, I found that many objects ar not really
>> deleted, but instead just marked deleted.
>> In fact, it's a general behavior in other projects (cinder, glance...).
>>
>> I understand that. It can be handy.
>>
>> But, is there a way to handle regular purging of theses elements inside
>> OpenStack configuration ?
>>
>> Or do we, operators, have to create batchs to purge all that ?
>>
>>
> I've found some blueprints indicating that there nothing for now, inside
> OpenStack :
>
> https://blueprints.launchpad.net/nova/+spec/purge-deleted-instances-cmd
> https://blueprints.launchpad.net/cinder/+spec/database-purge
> https://blueprints.launchpad.net/glance/+spec/database-purge
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[Gilles Mocellin]

Le 22/06/2016 à 23:26, Gilles Mocellin a écrit :

Hello,

While digging in nova's database, I found that many objects ar not 
really deleted, but instead just marked deleted.

In fact, it's a general behavior in other projects (cinder, glance...).

I understand that. It can be handy.

But, is there a way to handle regular purging of theses elements 
inside OpenStack configuration ?


Or do we, operators, have to create batchs to purge all that ?



I've found some blueprints indicating that there nothing for now, inside 
OpenStack :


https://blueprints.launchpad.net/nova/+spec/purge-deleted-instances-cmd
https://blueprints.launchpad.net/cinder/+spec/database-purge
https://blueprints.launchpad.net/glance/+spec/database-purge


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] How do you handle purge of database tables ?

[Silence Dogood]

I use thermite.

On Wed, Jun 22, 2016 at 5:26 PM, Gilles Mocellin <
gilles.mocel...@nuagelibre.org> wrote:

> Hello,
>
> While digging in nova's database, I found that many objects ar not really
> deleted, but instead just marked deleted.
> In fact, it's a general behavior in other projects (cinder, glance...).
>
> I understand that. It can be handy.
>
> But, is there a way to handle regular purging of theses elements inside
> OpenStack configuration ?
>
> Or do we, operators, have to create batchs to purge all that ?
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators