** Also affects: nova/rocky
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1804271
Title:
nova-api is broken in postgresql jobs
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) rocky series:
In Progress
Bug description:
We noticed in neutron-tempest-postgres-full periodic job that since
14.11.2018 it's failing because of issue in nova-api.
Example of failure are on
http://logs.openstack.org/periodic/git.openstack.org/openstack/neutron/master/neutron-tempest-postgres-full/1de7427/logs/screen-n-api.txt.gz?level=ERROR
Stacktrace:
Nov 20 06:34:20.578714 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi Traceback (most
recent call last):
Nov 20 06:34:20.578885 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 655, in wrapped
Nov 20 06:34:20.579046 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
f(*args, **kwargs)
Nov 20 06:34:20.579218 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.579379 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.579537 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.579737 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.579899 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.580070 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.580236 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.580396 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.580616 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.581036 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.581205 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.581364 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.581523 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.581693 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.581852 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.582008 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.582170 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.582336 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.582505 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.582663 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.582835 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.582987 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
func(*args, **kwargs)
Nov 20 06:34:20.583285 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 596, in create
Nov 20 06:34:20.583453 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi
**create_kwargs)
Nov 20 06:34:20.583651 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/hooks.py", line 154, in inner
Nov 20 06:34:20.583805 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi rv = f(*args,
**kwargs)
Nov 20 06:34:20.583957 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/compute/api.py", line 1737, in create
Nov 20 06:34:20.584136 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi
trusted_certs=trusted_certs)
Nov 20 06:34:20.584298 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/compute/api.py", line 1195, in _create_instance
Nov 20 06:34:20.584459 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi
supports_multiattach, network_metadata)
Nov 20 06:34:20.584623 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/compute/api.py", line 869, in _provision_instances
Nov 20 06:34:20.585034 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi context,
instance_type, min_count, max_count)
Nov 20 06:34:20.585199 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/compute/utils.py", line 967, in
check_num_instances_quota
Nov 20 06:34:20.585358 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi
check_user_id=user_id)
Nov 20 06:34:20.585530 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
184, in wrapper
Nov 20 06:34:20.585708 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi result =
fn(cls, context, *args, **kwargs)
Nov 20 06:34:20.585873 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/objects/quotas.py", line 339, in check_deltas
Nov 20 06:34:20.586037 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi **count_kwargs)
Nov 20 06:34:20.586195 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line
184, in wrapper
Nov 20 06:34:20.586363 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi result =
fn(cls, context, *args, **kwargs)
Nov 20 06:34:20.586520 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/objects/quotas.py", line 302, in count_as_dict
Nov 20 06:34:20.586677 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi context,
resource, *args, **kwargs)
Nov 20 06:34:20.586834 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/quota.py", line 1002, in count_as_dict
Nov 20 06:34:20.587002 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
res.count_as_dict(context, *args, **kwargs)
Nov 20 06:34:20.587160 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/quota.py", line 1178, in _instances_cores_ram_count
Nov 20 06:34:20.587324 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi context,
project_id)
Nov 20 06:34:20.587481 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/objects/cell_mapping.py", line 292, in
get_by_project_id
Nov 20 06:34:20.587721 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi db_mappings =
cls._get_by_project_id_from_db(context, project_id)
Nov 20 06:34:20.587984 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py",
line 1012, in wrapper
Nov 20 06:34:20.588247 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
fn(*args, **kwargs)
Nov 20 06:34:20.588493 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/opt/stack/new/nova/nova/objects/cell_mapping.py", line 283, in
_get_by_project_id_from_db
Nov 20 06:34:20.588745 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi
options(joinedload('cell_mapping', innerjoin=True)).\
Nov 20 06:34:20.588998 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2843, in
all
Nov 20 06:34:20.589248 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
list(self)
Nov 20 06:34:20.589509 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2995, in
__iter__
Nov 20 06:34:20.589783 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
self._execute_and_instances(context)
Nov 20 06:34:20.590043 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3018, in
_execute_and_instances
Nov 20 06:34:20.590316 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi result =
conn.execute(querycontext.statement, self._params)
Nov 20 06:34:20.590639 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 948,
in execute
Nov 20 06:34:20.590932 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
meth(self, multiparams, params)
Nov 20 06:34:20.591199 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 269,
in _execute_on_connection
Nov 20 06:34:20.591459 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi return
connection._execute_clauseelement(self, multiparams, params)
Nov 20 06:34:20.591746 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1060,
in _execute_clauseelement
Nov 20 06:34:20.592005 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi compiled_sql,
distilled_params
Nov 20 06:34:20.592273 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1200,
in _execute_context
Nov 20 06:34:20.592540 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi context)
Nov 20 06:34:20.592806 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1409,
in _handle_dbapi_exception
Nov 20 06:34:20.593101 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi
util.raise_from_cause(newraise, exc_info)
Nov 20 06:34:20.593558 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 265,
in raise_from_cause
Nov 20 06:34:20.593870 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi
reraise(type(exception), exception, tb=exc_tb, cause=cause)
Nov 20 06:34:20.594130 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1193,
in _execute_context
Nov 20 06:34:20.594398 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi context)
Nov 20 06:34:20.594661 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line
509, in do_execute
Nov 20 06:34:20.594937 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi
cursor.execute(statement, parameters)
Nov 20 06:34:20.595226 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi DBError:
(psycopg2.ProgrammingError) column "instance_mappings.created_at" must appear
in the GROUP BY clause or be used in an aggregate function
Nov 20 06:34:20.595488 ubuntu-xenial-inap-mtl01-0000583998
[email protected][24452]: ERROR nova.api.openstack.wsgi LINE 1: SELECT
instance_mappings.created_at AS instance_mappings_cre...
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1804271/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
