** Also affects: nova/pike
Importance: Undecided
Status: New
** Also affects: nova/queens
Importance: Undecided
Status: New
** Also affects: nova/rocky
Importance: Undecided
Status: New
** Changed in: nova/pike
Status: New => Triaged
** Changed in: nova/queens
Status: New => Triaged
** Changed in: nova/pike
Importance: Undecided => Medium
** Changed in: nova/rocky
Status: New => Triaged
** Changed in: nova/queens
Importance: Undecided => Medium
** Changed in: nova/rocky
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1815082
Title:
"DBNonExistentTable: (sqlite3.OperationalError) no such table:
services" when starting nova-metadata under uwsgi
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Compute (nova) pike series:
Triaged
Status in OpenStack Compute (nova) queens series:
Triaged
Status in OpenStack Compute (nova) rocky series:
Triaged
Bug description:
Jens Harbott reported this in devstack:
https://review.openstack.org/#/c/635519/
He's running the n-api-meta service in the subnode per the multinode
guide:
https://docs.openstack.org/devstack/latest/guides/multinode-lab.html
However with current devstack, which doesn't configure nova.conf with
database access on the subnode, n-api-meta fails with this:
http://paste.openstack.org/show/744683/
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: CRITICAL
nova [None req-538e6b11-f91b-48d8-9c7a-d177dff7739b None None] Unhandled error:
DBNonExistentTable: (sqlite3.OperationalError) no such t
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
Traceback (most recent call last):
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/bin/nova-metadata-wsgi", line 52, in <module>
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
application = init_application()
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/opt/stack/nova/nova/api/metadata/wsgi.py", line 20, in init_application
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
return wsgi_app.init_application(NAME)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 82, in
init_application
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
_setup_service(CONF.host, name)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 49, in
_setup_service
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
ctxt, host, binary)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py",
line 184, in wrapper
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
result = fn(cls, context, *args, **kwargs)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/opt/stack/nova/nova/objects/service.py", line 334, in
get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
host, binary)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/opt/stack/nova/nova/db/api.py", line 127, in
service_get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
return IMPL.service_get_by_host_and_binary(context, host, binary)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 242, in wrapped
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
return f(context, *args, **kwargs)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 500, in
service_get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
filter_by(binary=binary).\
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
2979, in first
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
ret = list(self[0:1])
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
2771, in __getitem__
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
return list(res)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
3081, in __iter__
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
return self._execute_and_instances(context)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
3106, in _execute_and_instances
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
result = conn.execute(querycontext.statement, self._params)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
980, in execute
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
return meth(self, multiparams, params)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py",
line 273, in _execute_on_connection
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
return connection._execute_clauseelement(self, multiparams, params)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
1099, in _execute_clauseelement
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
distilled_params,
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
1240, in _execute_context
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
e, statement, parameters, cursor, context
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
1456, in _handle_dbapi_exception
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
util.raise_from_cause(newraise, exc_info)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line
296, in raise_from_cause
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
reraise(type(exception), exception, tb=exc_tb, cause=cause)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line
1236, in _execute_context
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
cursor, statement, parameters, context
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
line 536, in do_execute
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
cursor.execute(statement, parameters)
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
DBNonExistentTable: (sqlite3.OperationalError) no such table: services [SQL:
u'SELECT services.created_at AS services_created_at, serv
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: ERROR nova
Feb 07 13:33:32 jh-devstack01a [email protected][22651]: unable to
load app 0 (mountpoint='') (callable not found or import error)
I believe this is due to the nova-metadata service not configuring the
conductor indirection API when run in uwsgi mode like it is when run
with eventlet:
https://github.com/openstack/nova/blob/3cfcd117ce7ddb5a8d94396191cde559db73fb86/nova/cmd/api_metadata.py#L45
It also appears that we don't have any multinode CI jobs that run with
the n-api-meta service configured so we don't hit this in the gate.
The workaround which most deployments probably use is configuring
nova.conf with database access and run n-api-meta with that, even from
subnodes.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1815082/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
