Reviewed: https://review.opendev.org/687922 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=83d7eb961dc8a5d4e815bc3f110f8523a532caeb Submitter: Zuul Branch: master
commit 83d7eb961dc8a5d4e815bc3f110f8523a532caeb Author: Rodolfo Alonso Hernandez <[email protected]> Date: Tue Oct 15 15:51:40 2019 +0000 Add OVS QoS driver cache for minimum bandwidth rules 1. Add OVS QoS driver cache Added minimum bandwidth rules cache in the OVS QoS driver. If a new port is detected in the integration bridge, managed by the OVS agent driver, the QoS extension is applied (if configured). If this new port does not have a QoS rule, by default the QoS configuration is reset in this port. Because this port has no Queue related, the call "QosOVSAgentDriver.delete_minimum_bandwidth" should not execute the rule cleanup. This QoS rule cache implementation per porti is currently used in the OVS QoS driver for the DSCP and the bandwitdh limit rules. 2. If a Queue cannot be deleted, log the QoS registers using it If a Queue is still used in a QoS register in the OVSDB, it cannot be deleted. With the current design, only one QoS rule is created in the database and several Queues are assigned to it. If something external to the OVS agent is handling the Queues and by mistake a Queue is assigned to other QoS registers, this error is now logged before raising the exception. Closes-Bug: #1845176 Change-Id: Ia9077fc20e4ca360819a2e368c8c1f9250e5a6d8 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1845176 Title: Removing of QoS queue in neutron-ovs-agent fails due to existing references Status in neutron: Fix Released Bug description: It happend in fullstack test: neutron.tests.fullstack.test_qos.TestMinBwQoSOvs.test_bw_limit_qos_port_removed(egress). Test failed and in neutron-openvswitch-agent logs we found error like: 2019-09-17 05:35:50.732 14622 DEBUG neutron.agent.common.ovs_lib [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Processing 1 OpenFlow rules. do_action_flows /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py:432 2019-09-17 05:35:50.733 14622 DEBUG neutron.agent.linux.utils [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running command (rootwrap daemon): ['ovs-ofctl', 'del-flows', '-O', 'OpenFlow10', 'br-intad6ecf25c', '-'] execute_rootwrap_daemon /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py:103 2019-09-17 05:35:50.750 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbClearCommand(table=QoS, record=0e930eb1-49eb-4f9e-8639-267abba5f4aa, column=queues) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2019-09-17 05:35:50.757 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbDestroyCommand(table=Queue, record=a1470780-1834-48d9-afd0-6fe41fcbb027) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84 2019-09-17 05:35:50.762 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"} 2019-09-17 05:35:50.764 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run txn.results.put(txn.do_commit()) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit raise RuntimeError(msg) RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"} 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error executing command: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"} 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command yield self._nested_txns_map[cur_thread_id] 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command KeyError: 140632266952480 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command During handling of the above exception, another exception occurred: 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command t.add(self) 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command del self._nested_txns_map[cur_thread_id] 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__ 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command raise result.ex 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"} 2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error while processing VIF ports: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"} 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent yield self._nested_txns_map[cur_thread_id] 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 140632266952480 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred: 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2435, in rpc_loop 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1968, in process_network_ports 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent devices_added_updated, provisioning_needed)) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1834, in treat_devices_added_or_updated 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.ext_manager.delete_port(self.context, {'port_id': device}) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/l2_agent_extensions_manager.py", line 54, in delete_port 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent extension.obj.delete_port(context, data) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 279, in delete_port 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._process_reset_port(port) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 299, in _process_reset_port 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.qos_driver.delete(port) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 94, in delete 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._handle_rule_delete(port, rule_type) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 119, in _handle_rule_delete 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent handler(port) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py", line 218, in delete_minimum_bandwidth 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.br_int.delete_minimum_bandwidth_queue(port['port_id']) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 985, in delete_minimum_bandwidth_queue 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._delete_queue(queue['_uuid']) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 1057, in _delete_queue 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.ovsdb.db_destroy('Queue', queue_id).execute(check_error=True) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent t.add(self) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent next(self.gen) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent del self._nested_txns_map[cur_thread_id] 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__ 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.result = self.commit() 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise result.ex 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent txn.results.put(txn.do_commit()) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise RuntimeError(msg) 2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"} Full log can be found at https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_847/681893/1/gate/neutron-fullstack/847f4d9/controller/logs/dsvm-fullstack-logs/TestMinBwQoSOvs.test_bw_limit_qos_port_removed_egress_/neutron-openvswitch-agent--2019-09-17--05-35-29-575569_log.txt.gz To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1845176/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
