Reviewed: https://review.openstack.org/230919 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8694c1619d774bb8a6c23ed4c0f33df2084849bc Submitter: Jenkins Branch: master
commit 8694c1619d774bb8a6c23ed4c0f33df2084849bc Author: Mark Goddard <[email protected]> Date: Thu Oct 1 17:37:45 2015 +0100 network: Don't repopulate instance info cache from Neutron ports Allocation of network interfaces for an instance can result in corruption of the instance info cache. The result is that the cache may contain duplicate entries for network interfaces. This can cause instance boot failure. This bug appears to be attributable to the combined effects of the fixes for bugs #1467581 and #1407664. This change reverts the fix for bug #1407664, whilst keeping a modified version of the unit test that was added with it. It also adds a second unit test. Change-Id: I53d5284907d44ae8b5546993f8fd461b385c39e6 Closes-bug: #1501735 Related-bug: #1467581 Related-bug: #1407664 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1501735 Title: Network interface allocation corrupts instance info cache Status in OpenStack Compute (nova): Fix Released Bug description: Allocation of network interfaces for an instance can result in corruption of the instance info cache in Nova. The result is that the cache may contain duplicate entries for network interfaces. This can cause failure to boot nodes, as seen with the Libvirt driver. Seen on Ubuntu / devstack / commit b0013d93ffeaed53bc28d9558def26bdb7041ed7. The issue can be reproduced using an instance with a large number of interfaces, for example using the heat stack in the attached YAML file heat-stack-many-interfaces.yaml. For improved reproducibility, add a short sleep in nova.network.neutronv2.api.API.allocate_for_instance, just before the call to self.get_instance_nw_info. This issue was found by SecurityFun23 when testing the fix for bug #1467581. The problem appears to be that in nova.network.neutronv2.api.API.allocate_for_instance, after the Neutron API calls to create/update ports, but before the instance info cache is updated in get_instance_nw_info, it is possible for another request to refresh the instance info cache. This will cause the new/updated ports to be added to the cache as they are discovered in Neutron. Then, the original request resumes, and unconditionally adds the new interfaces to the cache. This results in duplicate entries. The most likely candidate for another request is probably Neutron network-change notifications, which are triggered by the port update/create operation. The allocation of multiple interfaces is more likely to make the problem to occur, as Neutron API requests are made serially for each of the ports, allowing time for the notifications to arrive. The perceived problem in a more visual form: Request: - Allocate interfaces for an instance (nova.network.neutronv2.api.API.allocate_for_instance) - n x Neutron API port create/updates ------------------------------ Notification: - External event notification from Neutron - network-changed (nova.compute.manager.ComputeManager.external_instance_event) - Refresh instance network cache (network_api.get_instance_nw_info) - Query ports for device in Neutron - Add new ports to instance info cache ------------------------------- Request: - Refresh instance network cache with new interfaces (get_instance_nw_info) - Unconditionally add duplicate interfaces to cache. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1501735/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
