Reviewed: https://review.openstack.org/377933 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ceaf853894352b6d0ae12efe85ba5eb4e651e58a Submitter: Jenkins Branch: master
commit ceaf853894352b6d0ae12efe85ba5eb4e651e58a Author: Dan Smith <[email protected]> Date: Tue Sep 27 10:17:00 2016 -0700 Archive instance-related rows when the parent instance is deleted This is something I expect has been very broken for a long time. We have rows in tables such as instance_extra, instance_faults, etc that pertain to a single instance, and thus have a foreign key on their instance_uuid column that points to the instance. If any of those records exist, an instance can not be archived out of the main instances table. The archive routine currently "handles" this by skipping over said instances, and eventually iterating over all the tables to pull out any records that point to that instance, thus freeing up the instance itself for archival. The problem is, this only happens if those extra records are actually marked as deleted themselves. If we fail during a cleanup routine and leave some of them not marked as deleted, but where the instance they reference *is* marked as deleted, we will never archive them. This patch adds another phase of the archival process for any table that has an "instance_uuid" column, which attempts to archive records that point to these deleted instances. With this, using a very large real world sample database, I was able to archive my way down to zero deleted, un-archivable instances (from north of 100k). Closes-Bug: #1622545 Change-Id: I77255c77780f0c2b99d59a9c20adecc85335bb18 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1622545 Title: archive_deleted_rows isn't archiving instances Status in OpenStack Compute (nova): Fix Released Bug description: Running "nova-manage archive_deleted_rows ..." clears out little or none of the deleted nova instances for example running the command several times $ nova-manage --debug db archive_deleted_rows --max_rows 100000 --verbose I get +--------------------------+-------------------------+ | Table | Number of Rows Archived | +--------------------------+-------------------------+ | block_device_mapping | 10108 | | instance_actions | 31838 | | instance_actions_events | 2 | | instance_extra | 10108 | | instance_faults | 459 | | instance_info_caches | 10108 | | instance_metadata | 6037 | | instance_system_metadata | 17883 | | reservations | 9 | +--------------------------+-------------------------+ the only way I've been able to get an instances archived is to lower the --max-rows parameter, but this only deletes a small number of the instances and sometimes doesn't archive any at all In my nova-mange.log I have the following error 2016-09-12 09:22:21.658 17603 WARNING nova.db.sqlalchemy.api [-] IntegrityError detected when archiving table instances: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`nova`.`instance_extra`, CONSTRAINT `instance_extra_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))') [SQL: u'DELETE FROM instances WHERE instances.id in (SELECT T1.id FROM (SELECT instances.id \nFROM instances \nWHERE instances.deleted != %s ORDER BY instances.id \n LIMIT %s) as T1)'] [parameters: (0, 787)] mysql -e 'select count(*) from instances where deleted_at is not NULL;' nova +----------+ | count(*) | +----------+ | 70829 | +----------+ I'm running mitaka with this patch installed https://review.openstack.org/#/c/326730/1 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1622545/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
