Public bug reported: There is a problem - when I run tests in parallel then one/two can fail. As I see in logs one thread is deleting network while second thread is listing all ports. And second thread get exception 'NetworkNotFound'.
Part of neutron service logs is: 2015-12-18 06:29:05.151 INFO neutron.wsgi [req-4d303e7d-ae31-47b5-a644-552fceeb03ef user-0a50ad96 project-ce45a55a] 52.90.96.102 - - [18/Dec/2015 06:29:05] "DELETE /v2.0/networks/d2d2481a-4c20-452f-8088-6e6815694ac0.json HTTP/1.1" 204 173 0.426808 2015-12-18 06:29:05.173 ERROR neutron.policy [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] Policy check error while calling <bound method Ml2Plugin.get_network of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7f1ffffaa950>>! 2015-12-18 06:29:05.173 22048 ERROR neutron.policy Traceback (most recent call last): 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/policy.py", line 258, in __call__ 2015-12-18 06:29:05.173 22048 ERROR neutron.policy fields=[parent_field]) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 713, in get_network 2015-12-18 06:29:05.173 22048 ERROR neutron.policy result = super(Ml2Plugin, self).get_network(context, id, None) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 385, in get_network 2015-12-18 06:29:05.173 22048 ERROR neutron.policy network = self._get_network(context, id) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/db/db_base_plugin_common.py", line 188, in _get_network 2015-12-18 06:29:05.173 22048 ERROR neutron.policy raise n_exc.NetworkNotFound(net_id=id) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy NetworkNotFound: Network d2d2481a-4c20-452f-8088-6e6815694ac0 could not be found. 2015-12-18 06:29:05.173 22048 ERROR neutron.policy 2015-12-18 06:29:05.175 INFO neutron.api.v2.resource [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] index failed (client error): Network d2d2481a-4c20-452f-8088-6e6815694ac0 could not be found. 2015-12-18 06:29:05.175 INFO neutron.wsgi [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] 52.90.96.102 - - [18/Dec/2015 06:29:05] "GET /v2.0/ports.json?tenant_id=63f912ca152048c6a6b375784d90bd37 HTTP/1.1" 404 359 0.311871 Answer from Kevin Benton (in mailing list): Ah, I believe what is happening is that the network is being deleted after the port has been retrieved from the database during the policy check. The policy check retrieves the port's network to be able to enforce the network_owner lookup: https://github.com/openstack/neutron/blob/master/etc/policy.json#L6 So order of events seems to be: port list API call received ports retrieved from db network delete request is processed ports processed by policy engine policy engine triggers network lookup and hits 404 This appears to be a legitimate bug. Maybe we need to find a way to cache the network at port retrieval time for the policy engine. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1528031 Title: 'NetworkNotFound' exception during listing ports Status in neutron: New Bug description: There is a problem - when I run tests in parallel then one/two can fail. As I see in logs one thread is deleting network while second thread is listing all ports. And second thread get exception 'NetworkNotFound'. Part of neutron service logs is: 2015-12-18 06:29:05.151 INFO neutron.wsgi [req-4d303e7d-ae31-47b5-a644-552fceeb03ef user-0a50ad96 project-ce45a55a] 52.90.96.102 - - [18/Dec/2015 06:29:05] "DELETE /v2.0/networks/d2d2481a-4c20-452f-8088-6e6815694ac0.json HTTP/1.1" 204 173 0.426808 2015-12-18 06:29:05.173 ERROR neutron.policy [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] Policy check error while calling <bound method Ml2Plugin.get_network of <neutron.plugins.ml2.plugin.Ml2Plugin object at 0x7f1ffffaa950>>! 2015-12-18 06:29:05.173 22048 ERROR neutron.policy Traceback (most recent call last): 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/policy.py", line 258, in __call__ 2015-12-18 06:29:05.173 22048 ERROR neutron.policy fields=[parent_field]) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 713, in get_network 2015-12-18 06:29:05.173 22048 ERROR neutron.policy result = super(Ml2Plugin, self).get_network(context, id, None) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 385, in get_network 2015-12-18 06:29:05.173 22048 ERROR neutron.policy network = self._get_network(context, id) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy File "/opt/stack/neutron/neutron/db/db_base_plugin_common.py", line 188, in _get_network 2015-12-18 06:29:05.173 22048 ERROR neutron.policy raise n_exc.NetworkNotFound(net_id=id) 2015-12-18 06:29:05.173 22048 ERROR neutron.policy NetworkNotFound: Network d2d2481a-4c20-452f-8088-6e6815694ac0 could not be found. 2015-12-18 06:29:05.173 22048 ERROR neutron.policy 2015-12-18 06:29:05.175 INFO neutron.api.v2.resource [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] index failed (client error): Network d2d2481a-4c20-452f-8088-6e6815694ac0 could not be found. 2015-12-18 06:29:05.175 INFO neutron.wsgi [req-a406e696-6791-4345-8b04-215ca313ea67 user-0a50ad96 project-ce45a55a] 52.90.96.102 - - [18/Dec/2015 06:29:05] "GET /v2.0/ports.json?tenant_id=63f912ca152048c6a6b375784d90bd37 HTTP/1.1" 404 359 0.311871 Answer from Kevin Benton (in mailing list): Ah, I believe what is happening is that the network is being deleted after the port has been retrieved from the database during the policy check. The policy check retrieves the port's network to be able to enforce the network_owner lookup: https://github.com/openstack/neutron/blob/master/etc/policy.json#L6 So order of events seems to be: port list API call received ports retrieved from db network delete request is processed ports processed by policy engine policy engine triggers network lookup and hits 404 This appears to be a legitimate bug. Maybe we need to find a way to cache the network at port retrieval time for the policy engine. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1528031/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
