Reviewed: https://review.openstack.org/452351 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=032937ce51c64e0f5292ca3149e3b244863fffca Submitter: Jenkins Branch: master
commit 032937ce51c64e0f5292ca3149e3b244863fffca Author: Matt Riedemann <[email protected]> Date: Fri Mar 31 19:56:14 2017 -0400 Add regression test for quota decrement bug 1678326 This was spotted from someone validating the fix for bug 1670627. They reported that even though they failed to delete an instance in ERROR state that was in cell0, the quota usage was decremented. This is because we committed the quota reservation to decrement the usage before actually attempting to destroy the instance, rather than upon successful deletion. The rollback after InstanceNotFound is a noop because of how the Quotas.rollback method noops if the reservations were already committed. That is in itself arguably a bug, but not fixed here, especially since the counting quotas work in Pike will remove all of the reservations commit and rollback code. Change-Id: I12d1fa1a10f9014863123ac9cc3c63ddfb48378e Partial-Bug: #1678326 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1678326 Title: Quota is decremented during instance delete in cell0 even if the instance destroy fails Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: In Progress Bug description: This is a follow on from bug 1670627. As pointed out in comments 23 and 25, in the local delete case where there is no host and the instance is in a cell, we're decrementing quota even if the instance.destroy() operation fails. We commit the usage decrement here: https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1878 Attempt to destroy the instance here: https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1887 And if the instance.destroy() fails, we rollback the usage decrement here: https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1891 That rollback has no effect because once we commit a reservation, it's wiped out in the quotas object: https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/objects/quotas.py#L105 Attempting to rollback reservations on a quotas object that has already committed reservations is a noop: https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/objects/quotas.py#L111 -- Unlike the 'normal' (pre-cellsv2) local delete case which does the commit after the instance is destroyed: https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L2023 And we rollback (but not commit) if instance.destroy() fails because the instance is already deleted: https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L2028 -- The code in _delete() is wrong because it was copied from the code in _delete_while_booting() which is also wrong: https://github.com/openstack/nova/blob/88bc8dc5ce32748452c9d3acda9f35e77fedb6ce/nova/compute/api.py#L1784 So we have to fix both places. I suggest that we also change the noop behavior on the Quotas object such that if we attempt a reservation commit or rollback operation on a Quotas object that does not have any reservations, it is considered a programming error rather than a noop, because that's how this bug was introduced in the first place. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1678326/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
