Public bug reported: Got some db errors when call list servers and nova-api return http code 500
2019-07-02 10:08:39 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:39.736 16 INFO nova.osapi_compute.wsgi.server [req-5804b3dd-9968-481d-b03f-ed36192abb2f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 200 len: 42583 time: 0.2910399 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped. 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters context) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters result.read() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters packet_header = self._read_bytes(4) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters data = self._rfile.read(num_bytes) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters RuntimeError: reentrant call inside <_io.BufferedReader name=23> 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Unexpected exception in API method 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 208, in detail 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 359, in _get_servers 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1325, in fill_faults 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance_fault.py", line 108, in get_by_instance_uuids 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1906, in instance_fault_get_by_instance_uuids 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5999, in instance_fault_get_by_instance_uuids 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2613, in all 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions return list(self) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2761, in __iter__ 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions return self._execute_and_instances(context) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions close_with_result=True) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions **kw) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 893, in connection 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions execution_options=execution_options) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions engine, execution_options) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions conn = bind.contextual_connect() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions **kwargs) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__ 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self.dispatch.engine_connect(self, self.__branch) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions fn(*args, **kw) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions connection.scalar(select([1])) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions return self.execute(object, *multiparams, **params).scalar() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions return meth(self, multiparams, params) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions return connection._execute_clauseelement(self, multiparams, params) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions compiled_sql, distilled_params 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions context) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self._autorollback() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in _autorollback 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self._root._rollback_impl() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 670, in _rollback_impl 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self._handle_dbapi_exception(e, None, None, None, None) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in _handle_dbapi_exception 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions exc_info 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions reraise(type(exception), exception, tb=exc_tb, cause=cause) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 668, in _rollback_impl 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self.engine.dialect.do_rollback(self.connection) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions dbapi_connection.rollback() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self._read_ok_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 747, in _read_ok_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions pkt = self._read_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions packet_header = self._read_bytes(4) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions data = self._rfile.read(num_bytes) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions DBAPIError: (exceptions.RuntimeError) reentrant call inside <_io.BufferedReader name=23> 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Exception during reset or similar 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool Traceback (most recent call last): 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 636, in _finalize_fairy 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool fairy._reset(pool) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 776, in _reset 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool pool._dialect.do_rollback(self) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool dbapi_connection.rollback() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool self._read_ok_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 747, in _read_ok_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool pkt = self._read_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool packet_header = self._read_bytes(4) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool data = self._rfile.read(num_bytes) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool RuntimeError: reentrant call inside <_io.BufferedReader name=23> 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters [req-179abec8-6e3e-4ee2-a732-43709f25aa91 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped. 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters context) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters result.read() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 972, in _read_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters recv_data = self._read_bytes(bytes_to_read) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 986, in _read_bytes 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters self._sock.settimeout(self._read_timeout) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters AttributeError: 'NoneType' object has no attribute 'settimeout' 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.788 16 INFO nova.api.openstack.wsgi [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | <class 'sqlalchemy.exc.DBAPIError'> 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.789 16 INFO nova.osapi_compute.wsgi.server [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?host=node-4.domain.tld&project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 500 len: 568 time: 0.1924100 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1836029 Title: Got some db errors when call list servers Status in OpenStack Compute (nova): New Bug description: Got some db errors when call list servers and nova-api return http code 500 2019-07-02 10:08:39 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:39.736 16 INFO nova.osapi_compute.wsgi.server [req-5804b3dd-9968-481d-b03f-ed36192abb2f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 200 len: 42583 time: 0.2910399 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped. 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters context) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters result.read() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters packet_header = self._read_bytes(4) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters data = self._rfile.read(num_bytes) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters RuntimeError: reentrant call inside <_io.BufferedReader name=23> 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Unexpected exception in API method 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 208, in detail 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 359, in _get_servers 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1325, in fill_faults 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance_fault.py", line 108, in get_by_instance_uuids 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1906, in instance_fault_get_by_instance_uuids 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5999, in instance_fault_get_by_instance_uuids 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2613, in all 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions return list(self) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2761, in __iter__ 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions return self._execute_and_instances(context) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions close_with_result=True) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions **kw) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 893, in connection 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions execution_options=execution_options) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions engine, execution_options) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions conn = bind.contextual_connect() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions **kwargs) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__ 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self.dispatch.engine_connect(self, self.__branch) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions fn(*args, **kw) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions connection.scalar(select([1])) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions return self.execute(object, *multiparams, **params).scalar() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions return meth(self, multiparams, params) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions return connection._execute_clauseelement(self, multiparams, params) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions compiled_sql, distilled_params 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions context) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self._autorollback() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in _autorollback 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self._root._rollback_impl() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 670, in _rollback_impl 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self._handle_dbapi_exception(e, None, None, None, None) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in _handle_dbapi_exception 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions exc_info 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions reraise(type(exception), exception, tb=exc_tb, cause=cause) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 668, in _rollback_impl 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self.engine.dialect.do_rollback(self.connection) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions dbapi_connection.rollback() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions self._read_ok_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 747, in _read_ok_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions pkt = self._read_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions packet_header = self._read_bytes(4) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions data = self._rfile.read(num_bytes) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions DBAPIError: (exceptions.RuntimeError) reentrant call inside <_io.BufferedReader name=23> 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Exception during reset or similar 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool Traceback (most recent call last): 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 636, in _finalize_fairy 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool fairy._reset(pool) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 776, in _reset 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool pool._dialect.do_rollback(self) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool dbapi_connection.rollback() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool self._read_ok_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 747, in _read_ok_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool pkt = self._read_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool packet_header = self._read_bytes(4) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool data = self._rfile.read(num_bytes) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool RuntimeError: reentrant call inside <_io.BufferedReader name=23> 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters [req-179abec8-6e3e-4ee2-a732-43709f25aa91 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped. 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters context) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters result.read() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet() 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 972, in _read_packet 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters recv_data = self._read_bytes(bytes_to_read) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 986, in _read_bytes 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters self._sock.settimeout(self._read_timeout) 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters AttributeError: 'NoneType' object has no attribute 'settimeout' 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.788 16 INFO nova.api.openstack.wsgi [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | <class 'sqlalchemy.exc.DBAPIError'> 2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.789 16 INFO nova.osapi_compute.wsgi.server [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?host=node-4.domain.tld&project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 500 len: 568 time: 0.1924100 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1836029/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
