Public bug reported: When ML2/OVN backend is used there is a lot of errors like:
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, payload=payload) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 464, in _process_sg_rule_after_del_notification May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager self._ovn_client.delete_security_group_rule(context, sg_rule) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2414, in delete_security_group_rule May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager self._process_security_group_rule(rule, is_add_acl=False) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2402, in _process_security_group_rule May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager ovn_acl.update_acls_for_security_group( May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/common/ovn/acl.py", line 294, in update_acls_for_security_group May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager sg = plugin.get_security_group(admin_context, security_group_id) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 223, in wrapped May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager return f_with_retry(*args, **kwargs, May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 137, in wrapped May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager with excutils.save_and_reraise_exception(): May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager self.force_reraise() May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager raise self.value May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 135, in wrapped May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager return f(*args, **kwargs) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 144, in wrapper May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager with excutils.save_and_reraise_exception() as ectxt: May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager self.force_reraise() May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager raise self.value May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager return f(*args, **kwargs) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 183, in wrapped May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager with excutils.save_and_reraise_exception(): May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager self.force_reraise() May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager raise self.value May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 181, in wrapped May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager return f(*dup_args, **dup_kwargs) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 214, in get_security_group May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager ret = self._make_security_group_dict(self._get_security_group( May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 231, in _get_security_group May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager raise ext_sg.SecurityGroupNotFound(id=id) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager neutron.extensions.securitygroup.SecurityGroupNotFound: Security group caeb2144-af8c-4965-aa3b-3c93576f67a6 does not exist May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager May 18 17:48:46.908377 np0034069340 neutron-server[68119]: DEBUG neutron_lib.callbacks.manager [None req-f2eab262-f911-4632-b8be-f3914ead40b1 admin admin] Publish callbacks [] for security_group (None), before_response {{(pid=68119) _notify_loop /usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/manager.py:176}} May 18 17:48:46.909953 np0034069340 neutron-server[68119]: INFO neutron.wsgi [None req-f2eab262-f911-4632-b8be-f3914ead40b1 admin admin] 10.209.39.11,10.209.39.11 "DELETE /networking/v2.0/security-groups/caeb2144-af8c-4965-aa3b-3c93576f67a6 HTTP/1.1" status: 204 len: 173 time: 0.1885200 In the logs. For example in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5b3/883246/5/check/tempest- integrated-networking/5b39a98/controller/logs/screen-q-svc.txt but it's also in other jobs as well. ** Affects: neutron Importance: Low Status: Confirmed ** Tags: ovn -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2020437 Title: Deletion of SG caiuses stacktrace in the neutron server logs Status in neutron: Confirmed Bug description: When ML2/OVN backend is used there is a lot of errors like: May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, payload=payload) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 464, in _process_sg_rule_after_del_notification May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager self._ovn_client.delete_security_group_rule(context, sg_rule) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2414, in delete_security_group_rule May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager self._process_security_group_rule(rule, is_add_acl=False) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2402, in _process_security_group_rule May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager ovn_acl.update_acls_for_security_group( May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/common/ovn/acl.py", line 294, in update_acls_for_security_group May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager sg = plugin.get_security_group(admin_context, security_group_id) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 223, in wrapped May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager return f_with_retry(*args, **kwargs, May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 137, in wrapped May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager with excutils.save_and_reraise_exception(): May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager self.force_reraise() May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager raise self.value May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 135, in wrapped May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager return f(*args, **kwargs) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 144, in wrapper May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager with excutils.save_and_reraise_exception() as ectxt: May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager self.force_reraise() May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager raise self.value May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager return f(*args, **kwargs) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 183, in wrapped May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager with excutils.save_and_reraise_exception(): May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager self.force_reraise() May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager raise self.value May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 181, in wrapped May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager return f(*dup_args, **dup_kwargs) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 214, in get_security_group May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager ret = self._make_security_group_dict(self._get_security_group( May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 231, in _get_security_group May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager raise ext_sg.SecurityGroupNotFound(id=id) May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager neutron.extensions.securitygroup.SecurityGroupNotFound: Security group caeb2144-af8c-4965-aa3b-3c93576f67a6 does not exist May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager May 18 17:48:46.908377 np0034069340 neutron-server[68119]: DEBUG neutron_lib.callbacks.manager [None req-f2eab262-f911-4632-b8be-f3914ead40b1 admin admin] Publish callbacks [] for security_group (None), before_response {{(pid=68119) _notify_loop /usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/manager.py:176}} May 18 17:48:46.909953 np0034069340 neutron-server[68119]: INFO neutron.wsgi [None req-f2eab262-f911-4632-b8be-f3914ead40b1 admin admin] 10.209.39.11,10.209.39.11 "DELETE /networking/v2.0/security-groups/caeb2144-af8c-4965-aa3b-3c93576f67a6 HTTP/1.1" status: 204 len: 173 time: 0.1885200 In the logs. For example in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5b3/883246/5/check/tempest- integrated-networking/5b39a98/controller/logs/screen-q-svc.txt but it's also in other jobs as well. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2020437/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp