Public bug reported: Release: 2023.1 Driver: OVN
I'm seeing the following error in my logs. I don't know the exact trigger but seemed to occur during a tempest run: Error waiting on thread.: AssertionError: do not call blocking functions from the mainloop 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.dialect.do_execute( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup cursor.execute(statement, parameters) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = self._query(query) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn.query(q) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result.read() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup first_packet = self.connection._read_packet() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 692, in _read_packet 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup packet_header = self._read_bytes(4) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 732, in _read_bytes 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup data = self._rfile.read(num_bytes) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/usr/lib/python3.10/socket.py", line 705, in readinto 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._sock.recv_into(b) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 376, in recv_into 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 364, in _recv_loop 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._read_trampoline() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 332, in _read_trampoline 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._trampoline( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 211, in _trampoline 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return trampoline(fd, read=read, write=write, timeout=timeout, 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/__init__.py", line 141, in trampoline 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup assert hub.greenlet is not current, 'do not call blocking functions from the mainloop' 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred: 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.dialect.do_rollback(self.connection) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup dbapi_connection.rollback() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._execute_command(COMMAND.COM_QUERY, "ROLLBACK") 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise err.InterfaceError(0, "") 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pymysql.err.InterfaceError: (0, '') 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup The above exception was the direct cause of the following exception: 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2118, in _handle_dbapi_exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with util.safe_reraise(warn_only=True): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 84, in __exit__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup compat.raise_(value, with_traceback=traceback) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2119, in _handle_dbapi_exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._autorollback() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1187, in _autorollback 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._rollback_impl() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1059, in _rollback_impl 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._handle_dbapi_exception(e, None, None, None, None) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup util.raise_( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.dialect.do_rollback(self.connection) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup dbapi_connection.rollback() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._execute_command(COMMAND.COM_QUERY, "ROLLBACK") 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise err.InterfaceError(0, "") 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup sqlalchemy.exc.InterfaceError: (pymysql.err.InterfaceError) (0, '') 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup (Background on this error at: https://sqlalche.me/e/14/rvf5) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred: 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 268, in _perform_action_on_threads 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup action_func(x) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 342, in <lambda> 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup lambda x: x.wait(), 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 61, in wait 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.thread.wait() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._exit_event.wait() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/event.py", line 125, in wait 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = hub.switch() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.greenlet.switch() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 365, in run 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.wait(sleep_time) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/poll.py", line 80, in wait 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup presult = self.do_poll(seconds) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.poll.poll(seconds) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_hash_ring_db.remove_node_by_uuid( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 226, in wrapped 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f_with_retry(*args, **kwargs, 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception(): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*args, **kwargs) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception() as ectxt: 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*args, **kwargs) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception(): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*dup_args, **dup_kwargs) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/db/ovn_hash_ring_db.py", line 59, in remove_node_by_uuid 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_models.OVNHashRing.node_uuid == node_uuid).delete() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 3213, in delete 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = self.session.execute( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1711, in execute 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn = self._connection_for_bind(bind) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._transaction._connection_for_bind( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn = bind.connect() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._connection_cls(self, close_with_result=close_with_result) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 120, in __init__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.dispatch.engine_connect(self, _branch_from is not None) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup fn(*args, **kw) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 74, in _connect_ping_listener 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection.scalar(select(1)) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1262, in scalar 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.execute(object_, *multiparams, **params).scalar() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return connection._execute_clauseelement( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ret = self._execute_context( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._handle_dbapi_exception( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2137, in _handle_dbapi_exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.pool._invalidate(dbapi_conn_wrapper, e) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 283, in _invalidate 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection.invalidate(exception) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1090, in invalidate 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._checkin() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 982, in _checkin 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup _finalize_fairy( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 790, in _finalize_fairy 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection_record.checkin() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 531, in checkin 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pool.dispatch.checkin(connection, self) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup fn(*args, **kw) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 52, in _thread_yield 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup time.sleep(0) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 33, in sleep 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup assert hub.greenlet is not current, 'do not call blocking functions from the mainloop' 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Sorry for the long backtrace. The important bit seems to be: 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_hash_ring_db.remove_node_by_uuid( ** Affects: neutron Importance: Undecided Status: New ** Description changed: Release: 2023.1 Driver: OVN I'm seeing the following error in my logs. I don't know the exact trigger but seemed to occur during a tempest run: - Error waiting on thread.: AssertionError: do not call blocking functions from the mainloop - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.dialect.do_execute( - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup cursor.execute(statement, parameters) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = self._query(query) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn.query(q) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._affected_rows = self._read_query_result(unbuffered=unbuffered) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result.read() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup first_packet = self.connection._read_packet() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 692, in _read_packet - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup packet_header = self._read_bytes(4) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 732, in _read_bytes - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup data = self._rfile.read(num_bytes) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/usr/lib/python3.10/socket.py", line 705, in readinto - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._sock.recv_into(b) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 376, in recv_into - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 364, in _recv_loop - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._read_trampoline() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 332, in _read_trampoline - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._trampoline( - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 211, in _trampoline - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return trampoline(fd, read=read, write=write, timeout=timeout, - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/__init__.py", line 141, in trampoline - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup assert hub.greenlet is not current, 'do not call blocking functions from the mainloop' - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred: - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.dialect.do_rollback(self.connection) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup dbapi_connection.rollback() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._execute_command(COMMAND.COM_QUERY, "ROLLBACK") - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise err.InterfaceError(0, "") - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pymysql.err.InterfaceError: (0, '') - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup The above exception was the direct cause of the following exception: - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2118, in _handle_dbapi_exception - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with util.safe_reraise(warn_only=True): - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 84, in __exit__ - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup compat.raise_(value, with_traceback=traceback) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise exception - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2119, in _handle_dbapi_exception - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._autorollback() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1187, in _autorollback - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._rollback_impl() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1059, in _rollback_impl - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._handle_dbapi_exception(e, None, None, None, None) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup util.raise_( - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise exception - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.dialect.do_rollback(self.connection) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup dbapi_connection.rollback() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._execute_command(COMMAND.COM_QUERY, "ROLLBACK") - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise err.InterfaceError(0, "") - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup sqlalchemy.exc.InterfaceError: (pymysql.err.InterfaceError) (0, '') - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup (Background on this error at: https://sqlalche.me/e/14/rvf5) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred: - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 268, in _perform_action_on_threads - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup action_func(x) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 342, in <lambda> - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup lambda x: x.wait(), - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 61, in wait - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.thread.wait() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._exit_event.wait() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/event.py", line 125, in wait - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = hub.switch() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.greenlet.switch() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 365, in run - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.wait(sleep_time) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/poll.py", line 80, in wait - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup presult = self.do_poll(seconds) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.poll.poll(seconds) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_hash_ring_db.remove_node_by_uuid( - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 226, in wrapped - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f_with_retry(*args, **kwargs, - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception(): - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*args, **kwargs) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception() as ectxt: - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*args, **kwargs) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception(): - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*dup_args, **dup_kwargs) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/db/ovn_hash_ring_db.py", line 59, in remove_node_by_uuid - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_models.OVNHashRing.node_uuid == node_uuid).delete() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 3213, in delete - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = self.session.execute( - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1711, in execute - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn = self._connection_for_bind(bind) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._transaction._connection_for_bind( - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn = bind.connect() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._connection_cls(self, close_with_result=close_with_result) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 120, in __init__ - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.dispatch.engine_connect(self, _branch_from is not None) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__ - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup fn(*args, **kw) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 74, in _connect_ping_listener - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection.scalar(select(1)) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1262, in scalar - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.execute(object_, *multiparams, **params).scalar() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return connection._execute_clauseelement( - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ret = self._execute_context( - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._handle_dbapi_exception( - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2137, in _handle_dbapi_exception - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.pool._invalidate(dbapi_conn_wrapper, e) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 283, in _invalidate - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection.invalidate(exception) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1090, in invalidate - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._checkin() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 982, in _checkin - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup _finalize_fairy( - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 790, in _finalize_fairy - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection_record.checkin() - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 531, in checkin - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pool.dispatch.checkin(connection, self) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__ - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup fn(*args, **kw) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 52, in _thread_yield - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup time.sleep(0) - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 33, in sleep - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup assert hub.greenlet is not current, 'do not call blocking functions from the mainloop' - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup + Error waiting on thread.: AssertionError: do not call blocking functions from the mainloop + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.dialect.do_execute( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup cursor.execute(statement, parameters) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = self._query(query) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn.query(q) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._affected_rows = self._read_query_result(unbuffered=unbuffered) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result.read() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup first_packet = self.connection._read_packet() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 692, in _read_packet + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup packet_header = self._read_bytes(4) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 732, in _read_bytes + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup data = self._rfile.read(num_bytes) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/usr/lib/python3.10/socket.py", line 705, in readinto + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._sock.recv_into(b) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 376, in recv_into + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 364, in _recv_loop + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._read_trampoline() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 332, in _read_trampoline + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._trampoline( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 211, in _trampoline + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return trampoline(fd, read=read, write=write, timeout=timeout, + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/__init__.py", line 141, in trampoline + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup assert hub.greenlet is not current, 'do not call blocking functions from the mainloop' + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred: + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.dialect.do_rollback(self.connection) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup dbapi_connection.rollback() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._execute_command(COMMAND.COM_QUERY, "ROLLBACK") + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise err.InterfaceError(0, "") + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pymysql.err.InterfaceError: (0, '') + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup The above exception was the direct cause of the following exception: + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2118, in _handle_dbapi_exception + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with util.safe_reraise(warn_only=True): + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 84, in __exit__ + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup compat.raise_(value, with_traceback=traceback) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise exception + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2119, in _handle_dbapi_exception + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._autorollback() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1187, in _autorollback + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._rollback_impl() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1059, in _rollback_impl + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._handle_dbapi_exception(e, None, None, None, None) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup util.raise_( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise exception + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.dialect.do_rollback(self.connection) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup dbapi_connection.rollback() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._execute_command(COMMAND.COM_QUERY, "ROLLBACK") + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise err.InterfaceError(0, "") + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup sqlalchemy.exc.InterfaceError: (pymysql.err.InterfaceError) (0, '') + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup (Background on this error at: https://sqlalche.me/e/14/rvf5) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred: + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 268, in _perform_action_on_threads + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup action_func(x) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 342, in <lambda> + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup lambda x: x.wait(), + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 61, in wait + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.thread.wait() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._exit_event.wait() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/event.py", line 125, in wait + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = hub.switch() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.greenlet.switch() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 365, in run + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.wait(sleep_time) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/poll.py", line 80, in wait + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup presult = self.do_poll(seconds) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.poll.poll(seconds) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_hash_ring_db.remove_node_by_uuid( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 226, in wrapped + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f_with_retry(*args, **kwargs, + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception(): + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*args, **kwargs) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception() as ectxt: + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*args, **kwargs) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception(): + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*dup_args, **dup_kwargs) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/db/ovn_hash_ring_db.py", line 59, in remove_node_by_uuid + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_models.OVNHashRing.node_uuid == node_uuid).delete() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 3213, in delete + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = self.session.execute( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1711, in execute + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn = self._connection_for_bind(bind) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._transaction._connection_for_bind( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn = bind.connect() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._connection_cls(self, close_with_result=close_with_result) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 120, in __init__ + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.dispatch.engine_connect(self, _branch_from is not None) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__ + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup fn(*args, **kw) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 74, in _connect_ping_listener + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection.scalar(select(1)) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1262, in scalar + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.execute(object_, *multiparams, **params).scalar() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return connection._execute_clauseelement( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ret = self._execute_context( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._handle_dbapi_exception( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2137, in _handle_dbapi_exception + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.pool._invalidate(dbapi_conn_wrapper, e) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 283, in _invalidate + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection.invalidate(exception) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1090, in invalidate + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._checkin() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 982, in _checkin + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup _finalize_fairy( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 790, in _finalize_fairy + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection_record.checkin() + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 531, in checkin + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pool.dispatch.checkin(connection, self) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__ + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup fn(*args, **kw) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 52, in _thread_yield + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup time.sleep(0) + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 33, in sleep + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup assert hub.greenlet is not current, 'do not call blocking functions from the mainloop' + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Sorry for the long backtrace. The important bit seems to be: - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring - 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_hash_ring_db.remove_node_by_uuid( + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring + 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_hash_ring_db.remove_node_by_uuid( -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2058018 Title: AssertionError: do not call blocking functions from the mainloop Status in neutron: New Bug description: Release: 2023.1 Driver: OVN I'm seeing the following error in my logs. I don't know the exact trigger but seemed to occur during a tempest run: Error waiting on thread.: AssertionError: do not call blocking functions from the mainloop 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.dialect.do_execute( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup cursor.execute(statement, parameters) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = self._query(query) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn.query(q) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result.read() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup first_packet = self.connection._read_packet() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 692, in _read_packet 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup packet_header = self._read_bytes(4) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 732, in _read_bytes 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup data = self._rfile.read(num_bytes) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/usr/lib/python3.10/socket.py", line 705, in readinto 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._sock.recv_into(b) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 376, in recv_into 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 364, in _recv_loop 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._read_trampoline() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 332, in _read_trampoline 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._trampoline( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 211, in _trampoline 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return trampoline(fd, read=read, write=write, timeout=timeout, 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/__init__.py", line 141, in trampoline 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup assert hub.greenlet is not current, 'do not call blocking functions from the mainloop' 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred: 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.dialect.do_rollback(self.connection) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup dbapi_connection.rollback() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._execute_command(COMMAND.COM_QUERY, "ROLLBACK") 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise err.InterfaceError(0, "") 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pymysql.err.InterfaceError: (0, '') 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup The above exception was the direct cause of the following exception: 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2118, in _handle_dbapi_exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with util.safe_reraise(warn_only=True): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 84, in __exit__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup compat.raise_(value, with_traceback=traceback) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2119, in _handle_dbapi_exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._autorollback() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1187, in _autorollback 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._rollback_impl() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1059, in _rollback_impl 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._handle_dbapi_exception(e, None, None, None, None) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup util.raise_( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.dialect.do_rollback(self.connection) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup dbapi_connection.rollback() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._execute_command(COMMAND.COM_QUERY, "ROLLBACK") 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise err.InterfaceError(0, "") 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup sqlalchemy.exc.InterfaceError: (pymysql.err.InterfaceError) (0, '') 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup (Background on this error at: https://sqlalche.me/e/14/rvf5) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred: 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 268, in _perform_action_on_threads 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup action_func(x) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 342, in <lambda> 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup lambda x: x.wait(), 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 61, in wait 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.thread.wait() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._exit_event.wait() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/event.py", line 125, in wait 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = hub.switch() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.greenlet.switch() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 365, in run 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.wait(sleep_time) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/poll.py", line 80, in wait 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup presult = self.do_poll(seconds) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.poll.poll(seconds) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_hash_ring_db.remove_node_by_uuid( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 226, in wrapped 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f_with_retry(*args, **kwargs, 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception(): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*args, **kwargs) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception() as ectxt: 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*args, **kwargs) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup with excutils.save_and_reraise_exception(): 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.force_reraise() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup raise self.value 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return f(*dup_args, **dup_kwargs) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/db/ovn_hash_ring_db.py", line 59, in remove_node_by_uuid 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_models.OVNHashRing.node_uuid == node_uuid).delete() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 3213, in delete 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup result = self.session.execute( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1711, in execute 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn = self._connection_for_bind(bind) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._transaction._connection_for_bind( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup conn = bind.connect() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self._connection_cls(self, close_with_result=close_with_result) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 120, in __init__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.dispatch.engine_connect(self, _branch_from is not None) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup fn(*args, **kw) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 74, in _connect_ping_listener 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection.scalar(select(1)) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1262, in scalar 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return self.execute(object_, *multiparams, **params).scalar() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup return connection._execute_clauseelement( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ret = self._execute_context( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._handle_dbapi_exception( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2137, in _handle_dbapi_exception 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self.engine.pool._invalidate(dbapi_conn_wrapper, e) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 283, in _invalidate 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection.invalidate(exception) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1090, in invalidate 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup self._checkin() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 982, in _checkin 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup _finalize_fairy( 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 790, in _finalize_fairy 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup connection_record.checkin() 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 531, in checkin 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pool.dispatch.checkin(connection, self) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup fn(*args, **kw) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 52, in _thread_yield 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup time.sleep(0) 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 33, in sleep 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup assert hub.greenlet is not current, 'do not call blocking functions from the mainloop' 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Sorry for the long backtrace. The important bit seems to be: 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup ovn_hash_ring_db.remove_node_by_uuid( To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2058018/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
