Reviewed: https://review.openstack.org/273034 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=77de9653fd60a802b11f157972f7b3e81497e8a7 Submitter: Jenkins Branch: master
commit 77de9653fd60a802b11f157972f7b3e81497e8a7 Author: Kevin Benton <[email protected]> Date: Wed Jan 27 05:18:13 2016 -0800 Raise RetryRequest on policy parent not found During a port list operation, a port and its parent network may be concurrently deleted from the database after they have been retrieved from the DB but before policy is enforced. Then when the policy engine tries to do a get_network to check network ownership for a port on a network that no longer exists, it will encounter a NetworkNotFound exception from the core plugin. This exception was being propagated all of the way up to the whole API operation as a 404, which made no sense in the context of a port list. This patch adjusts the logic to catch any NotFound exceptions during this processing and convert them into a RetryRequest to trigger the API to restart the operation. At this point the objects will be gone from the database so the problematic items will not be passed to the policy engine for enforcement. Closes-Bug: #1528031 Change-Id: I89d12fe0767e1c7ecb68138b5f6f17aa68a68769 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1528031 Title: 'NetworkNotFound' exception during listing ports Status in neutron: Fix Released Bug description: There is a problem - when I run tests in parallel then one/two can fail. As I see in logs one thread is deleting network while second thread is listing all ports. And second thread get exception 'NetworkNotFound'. Part of neutron service logs is: 2015-12-18 06:29:05.151 INFO neutron.wsgi [req-4d303e7d-ae31-47b5-a644-552fceeb03ef user-0a50ad96 project-ce45a55a] 52.90.96.102 - - [18/Dec/2015 06:29:05] "DELETE /v2.0/networks/d2d2481a-4c20-452f-8088-6e6815694ac0.json HTTP/1.1" 204 173 0.426808 2015-12-18 06:29:05.173 ERROR neutron.policy [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] Policy check error while calling <bound method Ml2Plugin.get_network of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7f1ffffaa950>>! 2015-12-18 06:29:05.173 22048 ERROR neutron.policy Traceback (most recent call last): 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/policy.py", line 258, in __call__ 2015-12-18 06:29:05.173 22048 ERROR neutron.policy fields=[parent_field]) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 713, in get_network 2015-12-18 06:29:05.173 22048 ERROR neutron.policy result = super(Ml2Plugin, self).get_network(context, id, None) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 385, in get_network 2015-12-18 06:29:05.173 22048 ERROR neutron.policy network = self._get_network(context, id) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/db/db_base_plugin_common.py", line 188, in _get_network 2015-12-18 06:29:05.173 22048 ERROR neutron.policy raise n_exc.NetworkNotFound(net_id=id) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy NetworkNotFound: Network d2d2481a-4c20-452f-8088-6e6815694ac0 could not be found. 2015-12-18 06:29:05.173 22048 ERROR neutron.policy 2015-12-18 06:29:05.175 INFO neutron.api.v2.resource [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] index failed (client error): Network d2d2481a-4c20-452f-8088-6e6815694ac0 could not be found. 2015-12-18 06:29:05.175 INFO neutron.wsgi [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] 52.90.96.102 - - [18/Dec/2015 06:29:05] "GET /v2.0/ports.json?tenant_id=63f912ca152048c6a6b375784d90bd37 HTTP/1.1" 404 359 0.311871 Answer from Kevin Benton (in mailing list): Ah, I believe what is happening is that the network is being deleted after the port has been retrieved from the database during the policy check. The policy check retrieves the port's network to be able to enforce the network_owner lookup: https://github.com/openstack/neutron/blob/master/etc/policy.json#L6 So order of events seems to be: port list API call received ports retrieved from db network delete request is processed ports processed by policy engine policy engine triggers network lookup and hits 404 This appears to be a legitimate bug. Maybe we need to find a way to cache the network at port retrieval time for the policy engine. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1528031/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
