Reviewed: https://review.opendev.org/758472 Committed: https://git.openstack.org/cgit/openstack/ovn-octavia-provider/commit/?id=5b0715d96700d1c288f601b3ecb2cad507c2c134 Submitter: Zuul Branch: master
commit 5b0715d96700d1c288f601b3ecb2cad507c2c134 Author: Terry Wilson <[email protected]> Date: Thu Oct 15 12:53:36 2020 -0500 Fix leaked file descriptors by cleaning up objects A reference to the ovsdbapp Connection object is stored on the the API impl class and it is causing the object to stick around and its pipe fds to never close. Ultimately, python-ovs's call to select() is limited to 1024 file descriptors and will eventually error out. This deletes the connection and api objects explicitly when we are done with them. Change-Id: I97b27eaa293fb161724d34da88b4398f8b590c33 Co-authored-by: Brian Haley <[email protected]> Closes-Bug: #1894136 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1894136 Title: [OVN Octavia Provider] OVN provider fails during listener delete Status in neutron: Fix Released Bug description: The OVN provider is consistently failing during a listener delete as part of the member API tempest test tear down with a 'filedescriptor out of range in select()' error. o-api logs snippet: Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ValueError: filedescriptor out of range in select() Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn Traceback (most recent call last): Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 61, in start_connection Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn self.ovsdb_connection.start() Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout) Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 201, in wait_for_change Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn ovs_poller.block() Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 231, in block Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn events = self.poll.poll(self.timeout) Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 140, in poll Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn timeout) Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn ValueError: filedescriptor out of range in select() Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn Sep 03 15:44:05.172746 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR octavia.api.drivers.driver_factory [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] Unable to load provider driver ovn due to: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ERROR wsme.api [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] Server-side error: "Provider 'ovn' was not found.". Detail: Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: Traceback (most recent call last): Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 61, in start_connection Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: self.ovsdb_connection.start() Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: idlutils.wait_for_change(self.idl, self.timeout) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 201, in wait_for_change Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ovs_poller.block() Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 231, in block Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: events = self.poll.poll(self.timeout) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 140, in poll Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: timeout) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ValueError: filedescriptor out of range in select() Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: During handling of the above exception, another exception occurred: Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: Traceback (most recent call last): Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/opt/stack/octavia/octavia/api/drivers/driver_factory.py", line 44, in get_driver Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: invoke_on_load=True).driver Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/stevedore/driver.py", line 61, in __init__ Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: warn_on_missing_entrypoint=warn_on_missing_entrypoint Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/stevedore/named.py", line 81, in __init__ Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: verify_requirements) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 233, in _load_plugins Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: self._on_load_failure_callback(self, ep, err) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 225, in _load_plugins Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: verify_requirements, Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/stevedore/named.py", line 158, in _load_one_plugin Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: verify_requirements, Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 257, in _load_one_plugin Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: obj = plugin(*invoke_args, **invoke_kwds) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/driver.py", line 42, in __init__ Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: self._ovn_helper = ovn_helper.OvnProviderHelper() Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/helper.py", line 60, in __init__ Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: self.ovn_nbdb_api = self.ovn_nbdb.start() Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 174, in start Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: return impl_idl_ovn.OvsdbNbOvnIdl(self.conn) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 118, in __init__ Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: super(OvsdbNbOvnIdl, self).__init__(connection) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 57, in __init__ Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: super(Backend, self).__init__(connection) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 35, in __init__ Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: self.start_connection(connection) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 66, in start_connection Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: raise connection_exception Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct. Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: During handling of the above exception, another exception occurred: Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: Traceback (most recent call last): Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/usr/local/lib/python3.6/dist-packages/wsmeext/pecan.py", line 84, in callfunction Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: result = f(self, *args, **kwargs) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/opt/stack/octavia/octavia/api/v2/controllers/listener.py", line 632, in delete Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: driver = driver_factory.get_driver(provider) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: File "/opt/stack/octavia/octavia/api/drivers/driver_factory.py", line 49, in get_driver Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: raise exceptions.ProviderNotFound(prov=provider) Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found. Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: : octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found. Sep 03 15:44:05.178879 ubuntu-bionic-inap-mtl01-0019624710 [email protected][10087]: [pid: 10089|app: 0|req: 1109/2281] 198.72.124.16 () {56 vars in 1234 bytes} [Thu Sep 3 15:44:04 2020] DELETE /load-balancer/v2.0/lbaas/listeners/3537f411-5ab3-48c8-9669-c9fa1f40fee9 => generated 90 bytes in 447 msecs (HTTP/1.1 501) 4 headers in 169 bytes (1 switches on core 0) To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1894136/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
