Reviewed: https://review.openstack.org/385617 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d40322c7d4aa1dd6d595dfe415278c9f252f4da2 Submitter: Jenkins Branch: master
commit d40322c7d4aa1dd6d595dfe415278c9f252f4da2 Author: Swaminathan Vasudevan <[email protected]> Date: Fri Oct 7 10:30:40 2016 -0700 DVR: Fix race condition in creation of fip gateway In large-scale environments, we have seen a router update arrive for one tenant while we are still creating the router for a different tenant and initializing the shared floating IP gateway port. Sometimes these updates can get scheduled simultaneously, with the second running before we are done creating all the resources in the first, causing an exception when trying to set the default route since either the interface or IP address does not exist yet. Add a lock to better synchronize these functions so a create can finish before an update can be done. If it still fails, we will throw an exception so that the namespace will be cleaned-up and the update can be re-scheduled for the next iteration. Closes-Bug: #1631513 Change-Id: Ia8c92cea2f8798582c39ad3450ab3b3c45a356f7 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1631513 Title: DVR: Fix race conditions when trying to add default gateway for fip gateway port. Status in neutron: Fix Released Bug description: There seems to be a race condition when trying to add default gateway route in fip namespace for the fip agent gateway port. The way it happens is at high scale testing, when there is a router update that is currently happening for the Router-A which has a floatingip, a fip namespace is getting created and gateway ports plugged to the external bridge in the context of the fip namespace. While it is getting created, if there is another router update for the same Router-A, then it calls 'update-gateway-port' and tries to set the default gateway and fails. We do find a log message in the l3-agent with 'Failed to process compatible router' and also a TRACE in the l3-agent. Traceback (most recent call last): File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update self._process_router_if_compatible(router) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 440, in _process_router_if_compatible self._process_updated_router(router) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 454, in _process_updated_router ri.process(self) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 538, in process super(DvrLocalRouter, self).process(agent) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 31, in process super(DvrRouterBase, self).process(agent) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py", line 396, in call self.logger(e) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py", line 393, in call return func(*args, **kwargs) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 989, in process self.process_external(agent) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 491, in process_external self.create_dvr_fip_interfaces(ex_gw_port) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 522, in create_dvr_fip_interfaces self.fip_ns.update_gateway_port(fip_agent_port) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 243, in update_gateway_port ipd.route.add_gateway(gw_ip) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 690, in add_gateway self._as_root([ip_version], tuple(args)) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 361, in _as_root use_root_namespace=use_root_namespace) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 94, in _as_root log_fail_as_error=self.log_fail_as_error) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 103, in _execute log_fail_as_error=log_fail_as_error) File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute raise RuntimeError(msg) To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1631513/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
