Reviewed: https://review.opendev.org/c/openstack/neutron/+/781555 Committed: https://opendev.org/openstack/neutron/commit/90980f496cfa3cc5df1c93cf834a44f33d3f1f6f Submitter: "Zuul (22348)" Branch: master
commit 90980f496cfa3cc5df1c93cf834a44f33d3f1f6f Author: Terry Wilson <[email protected]> Date: Thu Mar 18 15:01:07 2021 -0500 Add wait for the post-fork event to nb/sb objects It is possible for events from the nb/sb to fire before the opposite db connection is made. These events can call back into driver code which tries to access the other db before it is connected. Closes-Bug: #1929197 Closes-Bug: #1928794 Closes-Bug: #1929633 Change-Id: If947581b90ced42981c4611c32de8f428a052c69 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1928794 Title: [OVN] DHCP agent fails to update network Status in neutron: Fix Released Bug description: In latest Victoria build, DHCP agent is unable to update networks with OVN driver: 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent [req-e8c3cd96-2a53-4ea8-b2eb-4de64af3600c - - - - -] Unable to enable dhcp for c9caf247-edb4-4282-a253-b948de5f9da7.: oslo_messaging.rpc.client.RemoteError: Remote error: MechanismDriverError ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 311, in update_dhcp_port\n return self._port_action(plugin, context, port, \'update_port\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 101, in _port_action\n return plugin.update_port(context, port[\'id\'], port)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 681, in inner\n return f(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped\n return method(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1728, in update_port\n self.mechanism_manager.update_port_precommit(mech_context)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 731, in update_port_precommit\n raise_db_retriable=True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 495, in _call_on_drivers\n errors=errors\n', 'neutron.plugins.ml2.common.exceptions.MechanismDriverError\n']. 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 227, in call_driver 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent rv = getattr(driver, action)(**action_kwargs) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 266, in enable 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent common_utils.wait_until_true(self._enable, timeout=300) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 697, in wait_until_true 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent while not predicate(): 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 278, in _enable 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1676, in setup 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent self._cleanup_stale_devices(network, dhcp_port=None) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent self.force_reraise() 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent six.reraise(self.type_, self.value, self.tb) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent raise value 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1671, in setup 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent port = self.setup_dhcp_port(network) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1610, in setup_dhcp_port 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent dhcp_port = setup_method(network, device_id, dhcp_subnets) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 1534, in _setup_reserved_dhcp_port 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent 'device_id': device_id}}) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 894, in update_dhcp_port 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent port_id=port_id, port=port, host=self.host) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/rpc.py", line 157, in call 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent return self._original_context.call(ctxt, method, **kwargs) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent transport_options=self.transport_options) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent transport_options=transport_options) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 682, in send 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent transport_options=transport_options) 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 672, in _send 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent raise result 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent oslo_messaging.rpc.client.RemoteError: Remote error: MechanismDriverError 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 311, in update_dhcp_port\n return self._port_action(plugin, context, port, \'update_port\')\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 101, in _port_action\n return plugin.update_port(context, port[\'id\'], port)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 681, in inner\n return f(self, context, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped\n return method(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped\n LOG.debug("Retry wrapper got retriable exception: %s", e)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped\n return f(*dup_args, **dup_kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1728, in update_port\n self.mechanism_manager.update_port_precommit(mech_context)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 731, in update_port_precommit\n raise_db_retriable=True)\n', ' File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 495, in _call_on_drivers\n errors=errors\n', 'neutron.plugins.ml2.common.exceptions.MechanismDriverError\n']. 2021-05-18 11:42:32.508 7 ERROR neutron.agent.dhcp.agent Neutron: 2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers [req-e8c3cd96-2a53-4ea8-b2eb-4de64af3600c - - - - -] Mechanism driver 'ovn' failed in update_port_precommit: AttributeError: 'NoneType' object has no attribute 'chassis_exists' 2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers Traceback (most recent call last): 2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers 2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context) 2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 695, in update_port_precommit 2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers context.original_host): 2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 616, in _is_port_provisioning_required 2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers if not self._sb_ovn.chassis_exists(host): 2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers AttributeError: 'NoneType' object has no attribute 'chassis_exists' 2021-05-18 11:42:32.500 29 ERROR neutron.plugins.ml2.managers 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server [req-e8c3cd96-2a53-4ea8-b2eb-4de64af3600c - - - - -] Exception during message handling: neutron.plugins.ml2.common.exceptions.MechanismDriverError 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return func(*args, **kwargs) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return func(*args, **kwargs) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 311, in update_dhcp_port 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return self._port_action(plugin, context, port, 'update_port') 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 101, in _port_action 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return plugin.update_port(context, port['id'], port) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 681, in inner 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return method(*args, **kwargs) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise value 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/plugin.py", line 1728, in update_port 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server self.mechanism_manager.update_port_precommit(mech_context) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 731, in update_port_precommit 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server raise_db_retriable=True) 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 495, in _call_on_drivers 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server errors=errors 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server neutron.plugins.ml2.common.exceptions.MechanismDriverError 2021-05-18 11:42:32.503 29 ERROR oslo_messaging.rpc.server This problem does not exist in earlier builds (>1 month) of Neutron server. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1928794/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
