On 16 December 2016 at 21:40, Ben Nemec <openst...@nemebean.com> wrote: > Just a heads up for everyone, I've added some DB cleanup jobs to rh1 which > will hopefully prevent the performance degradations over time that we've > been seeing in that environment. Specifically, the crontab now looks like > this: > > # Clean up heat db > 0 5 * * * heat-manage purge_deleted 7 > # Archive nova db entries 5 times so we get everything > 0,10,20,30,40 6 * * * nova-manage db archive_deleted_rows --max_rows 100000 >
lgtm, We did a number of things last week in order to deal with the performance problems, these db archive commands cover part of it and needed to be added. The other thing we need to keep an eye on is the number of OVS ports on each compute node, on some compute nodes we had over 2000 ovs ports and ovs couldn't be restarted(and some other services). Ultimately to deal with this we rebooted each compute node and allowed neutron-ovs-cleanup to delete the unused ports. This wasn't ideal because 1) neutron-ovs-cleanup in some cases took a long time to delete the ports, and delayed nova-compute from starting up, so compute nodes couldn't be used for a extended period of time. 2) This also caused a reboot of the infrastructure nodes we run on rh1 (e.g. proxy, te-broker etc...) some of these didn't come back as expected, where needed patches have been submitted[1][2] I haven't done anything to prevent the ports building up again, so this is probably still a ongoing issue we need to look into [1] - https://review.openstack.org/#/c/409930/ [2] - https://review.openstack.org/#/c/406927/ > I picked 5 and 6 AM UTC because I think that's before most people in the EU > are starting and well after the US is done so the cloud should be pretty > quiet at that time. > > I think it's worth noting that we should probably be setting up this sort of > thing on initial deployment by default. Maybe we are now (rh1 is still back > on Mitaka), but if not we should figure out some appropriate defaults. > > -Ben > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev