There is very little information in this bug report, like what release
of nova are you using? What's the env topology? There is obviously a
mysql connection failure, so you should probably investigate the mysql
deployment / config / logs as that's what nova is failing on:
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions raise
err.OperationalError(2006, "MySQL server has gone away (%r)" % (e,))
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone
away (error(32, 'Broken pipe'))")
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1642234
Title:
error nova.api.openstack.extensions
Status in OpenStack Compute (nova):
Invalid
Bug description:
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
[req-3a504b21-c9c9-42e4-b246-9b506f47989b 33c709637a53464c9b459a80a64395c2
804207cd6c25496f92fa0f1d1c17d517 - - -] DBAPIError exception wrapped from
(pymysql.err.InternalError) (1927, u'Connection was killed') [SQL: u'SELECT 1']
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters Traceback
(most recent call last):
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in
_execute_context
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters context)
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in
do_execute
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
cursor.execute(statement, parameters)
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters result
= self._query(query)
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
conn.query(q)
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 781, in query
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 942, in
_read_query_result
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
result.read()
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1138, in read
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
first_packet = self.connection._read_packet()
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in
_read_packet
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
packet.check_error()
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in
check_error
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
err.raise_mysql_exception(self._data)
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in
raise_mysql_exception
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
_check_mysql_exception(errinfo)
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters File
"/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in
_check_mysql_exception
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters raise
InternalError(errno, errorvalue)
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
InternalError: (1927, u'Connection was killed')
2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
[req-3a504b21-c9c9-42e4-b246-9b506f47989b 33c709637a53464c9b459a80a64395c2
804207cd6c25496f92fa0f1d1c17d517 - - -] Unexpected exception in API method
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions Traceback
(most recent call last):
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478,
in wrapped
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line
194, in index
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
_services = self._get_services_list(req)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line
87, in _get_services_list
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
_services = self._get_services(req)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line
49, in _get_services
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions for s in
self.host_api.service_get_all(context, set_zones=True)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3717, in
service_get_all
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
set_zones=set_zones)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 181, in
wrapper
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions result =
fn(cls, context, *args, **kwargs)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/objects/service.py", line 406, in get_all
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
db_services = db.service_get_all(context, disabled=disabled)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/db/api.py", line 136, in service_get_all
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions return
IMPL.service_get_all(context, disabled)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 315, in
wrapped
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions return
f(context, *args, **kwargs)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 527, in
service_get_all
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions return
query.all()
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2588, in all
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions return
list(self)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in
__iter__
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions return
self._execute_and_instances(context)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in
_execute_and_instances
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
close_with_result=True)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2740, in
_connection_from_session
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions **kw)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 905, in
connection
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
execution_options=execution_options)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in
_connection_for_bind
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions engine,
execution_options)2016-11-16 19:44:18.958 2762 ERROR
nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in
_connection_for_bind
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions engine,
execution_options)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in
_connection_for_bind
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions conn =
bind.contextual_connect()
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in
contextual_connect
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions **kwargs)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in
__init__
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
self.dispatch.engine_connect(self, self.__branch)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in
__call__
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
fn(*args, **kw)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 72, in
_connect_ping_listener
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
connection.scalar(select([1]))
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in
scalar
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions return
self.execute(object, *multiparams, **params).scalar()
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in
execute
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions return
meth(self, multiparams, params)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in
_execute_on_connection
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions return
connection._execute_clauseelement(self, multiparams, params)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in
_execute_clauseelement
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
compiled_sql, distilled_params
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in
_execute_context
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions context)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in
_handle_dbapi_exception
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
self._autorollback()
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in
_autorollback
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
self._root._rollback_impl()
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 670, in
_rollback_impl
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
self._handle_dbapi_exception(e, None, None, None, None)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in
_handle_dbapi_exception
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions exc_info
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in
raise_from_cause
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
reraise(type(exception), exception, tb=exc_tb)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 668, in
_rollback_impl
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
self.engine.dialect.do_rollback(self.connection)
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line
2526, in do_rollback
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
dbapi_connection.rollback()
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 723, in rollback
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 970, in
_execute_command
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
self._write_bytes(prelude + sql[:chunk_size-1])
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/pymysql/connections.py", line 929, in
_write_bytes
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions raise
err.OperationalError(2006, "MySQL server has gone away (%r)" % (e,))
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone
away (error(32, 'Broken pipe'))")
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
2016-11-16 19:44:18.984 2762 INFO nova.api.openstack.wsgi
[req-3a504b21-c9c9-42e4-b246-9b506f47989b 33c709637a53464c9b459a80a64395c2
804207cd6c25496f92fa0f1d1c17d517 - - -] HTTP exception thrown: Unexpected API
Error. Please report this at http://bugs.launchpad.net/nova/ and attach the
Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'>
2016-11-16 19:44:18.985 2762 INFO nova.osapi_compute.wsgi.server
[req-3a504b21-c9c9-42e4-b246-9b506f47989b 33c709637a53464c9b459a80a64395c2
804207cd6c25496f92fa0f1d1c17d517 - - -] 192.168.153.130 "GET
/v2.1/804207cd6c25496f92fa0f1d1c17d517/os-services HTTP/1.1" status: 500 len:
507 time: 0.7481852
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1642234/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
