Public bug reported: When deleting a cluster with several networks and subnets, the following error was found when executing the shell command 'openstack subnet list':
019-08-06 08:26:10.000 28 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200 len: 248 time: 0.0012720 2019-08-06 08:26:10.803 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.15', 41484) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] GET failed.: AttributeError: 'Subnet' object has no attribute '_obj_network_id' 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 131, in index 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return self.get(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 141, in get 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation **query_params)} 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1075, in get_subnets 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation for subnet_object in subnet_objs 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 134, in _make_subnet_dict 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation 'network_id': subnet['network_id'], 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 774, in __getitem__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, name) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 68, in getter 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, attrname) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'Subnet' object has no attribute '_obj_network_id' 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation 2019-08-06 08:26:11.523 28 INFO neutron.wsgi [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] 172.17.1.15 "GET /v2.0/subnets HTTP/1.1" status: 500 len: 344 time: 0.7188911 2019-08-06 08:26:12.002 34 DEBUG neutron.wsgi [-] (34) accepted ('172.17.1.15', 41734) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883 2019-08-06 08:26:12.004 34 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200 len: 248 time: 0.0012681 Which means that some subnets where left behind without an associated network. In this case ns/openshift-kube-controller-manager-subnet: MariaDB [ovs_neutron]> select * from subnets; +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+ | project_id | id | name | network_id | ip_version | cidr | gateway_ip | enable_dhcp | ipv6_ra_mode | ipv6_address_mode | subnetpool_id | standard_attr_id | segment_id | +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+ | 755fea49ab0643d0b731073e87e739e0 | 26953b3b-fa6c-46a0-9e60-6a7d5cadf580 | lb-mgmt-subnet | 4bfa167b-fcf9-4ad8-b5f3-99c6141b20a4 | 4 | 172.24.0.0/16 | 172.24.0.1 | 1 | NULL | NULL | NULL | 8 | NULL | | dcb34bf98dea46928294ff6762d9c167 | 5ea30bc8-6eb1-4494-96e6-b37d9a70fb09 | ns/openshift-kube-controller-manager-subnet | NULL | 4 | 10.128.34.0/24 | 10.128.34.1 | 0 | NULL | NULL | cc48743c-e44c-48ee-b29c-5ceb53f60fbd | 9493 | NULL | | 755fea49ab0643d0b731073e87e739e0 | 8f4a9baf-ad79-4730-9449-671706f24d7a | external_subnet | 004f76a6-3867-46f4-8718-5498634945a3 | 4 | 10.0.0.0/24 | 10.0.0.1 | 0 | NULL | NULL | NULL | 27 | NULL | +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+ This is only happening rarely, not 100% of the time. ** Affects: neutron Importance: Undecided Assignee: Rodolfo Alonso (rodolfo-alonso-hernandez) Status: New ** Changed in: neutron Assignee: (unassigned) => Rodolfo Alonso (rodolfo-alonso-hernandez) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1839658 Title: "subnet" register in the DB can have network_id=NULL Status in neutron: New Bug description: When deleting a cluster with several networks and subnets, the following error was found when executing the shell command 'openstack subnet list': 019-08-06 08:26:10.000 28 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200 len: 248 time: 0.0012720 2019-08-06 08:26:10.803 28 DEBUG neutron.wsgi [-] (28) accepted ('172.17.1.15', 41484) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] GET failed.: AttributeError: 'Subnet' object has no attribute '_obj_network_id' 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 131, in index 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return self.get(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 141, in get 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation **query_params)} 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1075, in get_subnets 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation for subnet_object in subnet_objs 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py", line 134, in _make_subnet_dict 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation 'network_id': subnet['network_id'], 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 774, in __getitem__ 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, name) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 68, in getter 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation return getattr(self, attrname) 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'Subnet' object has no attribute '_obj_network_id' 2019-08-06 08:26:11.520 28 ERROR neutron.pecan_wsgi.hooks.translation 2019-08-06 08:26:11.523 28 INFO neutron.wsgi [req-d757636f-b963-46c1-989e-75dc8fb8406c 404ce30e61e64ce7bc67db81b312c24e dcb34bf98dea46928294ff6762d9c167 - default default] 172.17.1.15 "GET /v2.0/subnets HTTP/1.1" status: 500 len: 344 time: 0.7188911 2019-08-06 08:26:12.002 34 DEBUG neutron.wsgi [-] (34) accepted ('172.17.1.15', 41734) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883 2019-08-06 08:26:12.004 34 INFO neutron.wsgi [-] 172.17.1.15 "OPTIONS / HTTP/1.0" status: 200 len: 248 time: 0.0012681 Which means that some subnets where left behind without an associated network. In this case ns/openshift-kube-controller-manager-subnet: MariaDB [ovs_neutron]> select * from subnets; +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+ | project_id | id | name | network_id | ip_version | cidr | gateway_ip | enable_dhcp | ipv6_ra_mode | ipv6_address_mode | subnetpool_id | standard_attr_id | segment_id | +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+ | 755fea49ab0643d0b731073e87e739e0 | 26953b3b-fa6c-46a0-9e60-6a7d5cadf580 | lb-mgmt-subnet | 4bfa167b-fcf9-4ad8-b5f3-99c6141b20a4 | 4 | 172.24.0.0/16 | 172.24.0.1 | 1 | NULL | NULL | NULL | 8 | NULL | | dcb34bf98dea46928294ff6762d9c167 | 5ea30bc8-6eb1-4494-96e6-b37d9a70fb09 | ns/openshift-kube-controller-manager-subnet | NULL | 4 | 10.128.34.0/24 | 10.128.34.1 | 0 | NULL | NULL | cc48743c-e44c-48ee-b29c-5ceb53f60fbd | 9493 | NULL | | 755fea49ab0643d0b731073e87e739e0 | 8f4a9baf-ad79-4730-9449-671706f24d7a | external_subnet | 004f76a6-3867-46f4-8718-5498634945a3 | 4 | 10.0.0.0/24 | 10.0.0.1 | 0 | NULL | NULL | NULL | 27 | NULL | +----------------------------------+--------------------------------------+---------------------------------------------+--------------------------------------+------------+----------------+-------------+-------------+--------------+-------------------+--------------------------------------+------------------+------------+ This is only happening rarely, not 100% of the time. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1839658/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp