*** This bug is a duplicate of bug 1722404 ***
https://bugs.launchpad.net/bugs/1722404
This was fixed in queens and backported to stable/pike, but we'd need to
also backport https://review.openstack.org/#/c/512456/ to stable/ocata
(newton is end of life upstream).
** This bug has been marked a duplicate of bug 1722404
Database transactions can fail with "TypeError: Can't upgrade a READER
transaction to a WRITER mid-transaction" because of scatter_gather_cells
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1746509
Title:
TypeError: Can't upgrade a READER transaction to a WRITER mid-
transaction
Status in OpenStack Compute (nova):
New
Bug description:
Hi, I was running OPenstack Newton with no nova_cell0 database and
placement-api setup . After migrate to Openstack Pike and correctly setup the
nova_cell0 and placement-api everything is working fine except the openstack
server list on tenant that already exist .
For example :
1. For a new tenant created after the migration at Pike.
nova --os-project-name="New Project" list
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
| ID | Name | Status | Task
State | Power State | Networks |
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
| c41c7e8d-4bc0-4a0f-a9d3-dc719ae2aff0 | SAMPLE_VM | ACTIVE | -
| Running | SAMPLE-SUBNET=192.168.0.8 |
| 3d3d3e10-f326-4a92-9253-1511e738d1cc | SECOND_INSTANCE | ACTIVE | -
| Running | SAMPLE-SUBNET=192.168.0.5 |
+--------------------------------------+-----------------+--------+------------+-------------+---------------------------+
2. For a old tenant created before the migration at Newton .
nova --os-project-name="InterCon" list
ERROR (ClientException): Unexpected API Error. Please report this at
http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> (HTTP 500) (Request-ID:
req-0dd4ef4d-54c2-4cfd-b8d9-636c5736ef5f)
And here the log related to this error .
2018-01-31 07:45:35.832 2340 DEBUG nova.compute.api
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] Searching by: {'deleted': False} get_all
/usr/lib/python2.7/site-packages/nova/compute/api.py:2311
2018-01-31 07:45:35.837 2340 DEBUG oslo_concurrency.lockutils
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] Lock "00000000-0000-0000-0000-000000000000"
acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited
0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-31 07:45:35.837 2340 DEBUG oslo_concurrency.lockutils
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] Lock "00000000-0000-0000-0000-000000000000"
released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held
0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-31 07:45:35.858 2340 DEBUG nova.compute.api
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] Skipping already-collected cell0 list
_get_instances_by_filters_all_cells
/usr/lib/python2.7/site-packages/nova/compute/api.py:2503
2018-01-31 07:45:35.858 2340 DEBUG nova.compute.api
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] Listing 1000 instances in cell
ea0a32fb-c148-4d4f-a42d-4e46356ff926 _get_instances_by_filters_all_cells
/usr/lib/python2.7/site-packages/nova/compute/api.py:2506
2018-01-31 07:45:35.859 2340 DEBUG oslo_concurrency.lockutils
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] Lock "ea0a32fb-c148-4d4f-a42d-4e46356ff926"
acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited
0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2018-01-31 07:45:35.859 2340 DEBUG oslo_concurrency.lockutils
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] Lock "ea0a32fb-c148-4d4f-a42d-4e46356ff926"
released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held
0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2018-01-31 07:45:35.934 2340 DEBUG nova.objects.service
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] Generated UUID
f9ae1e59-21be-4fc3-9b02-ebb4c35783ec for service 19 _from_db_object
/usr/lib/python2.7/site-packages/nova/objects/service.py:245
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] Unexpected exception in API method:
TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions Traceback
(most recent call last):
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 336,
in wrapped
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181,
in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return
func(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181,
in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return
func(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line
168, in detail
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions servers
= self._get_servers(req, is_detail=True)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line
311, in _get_servers
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2415, in get_all
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2512, in
_get_instances_by_filters_all_cells
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2543, in
_get_instances_by_filters
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
expected_attrs=fields, sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in
wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions result =
fn(cls, context, *args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1245, in
get_by_filters
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
use_slave=use_slave, sort_keys=sort_keys, sort_dirs=sort_dirs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in
wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1235, in
_get_by_filters_impl
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
expected_attrs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1198, in
_make_instance_list
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
expected_attrs=expected_attrs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 448, in
_from_db_object
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
objects.Service, db_inst['services'])
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 1121, in
obj_make_list
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
**extra_args)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/objects/service.py", line 246, in
_from_db_object
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
service.save()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in
wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return
fn(self, *args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/objects/service.py", line 363, in save
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
db_service = db.service_update(self._context, self.id, updates)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return
IMPL.service_update(context, service_id, values)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
ectxt.value = e.inner_exc
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
self.force_reraise()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
six.reraise(self.type_, self.value, self.tb)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return
f(*args, **kwargs)
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 249, in
wrapped
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions with
ctxt_mgr.writer.using(context):
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions return
self.gen.next()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
1034, in _transaction_scope
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
context=context) as resource:
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
684, in _produce_block
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
self._writer()
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions File
"/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
702, in _writer
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions "Can't
upgrade a READER transaction "
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions TypeError:
Can't upgrade a READER transaction to a WRITER mid-transaction
2018-01-31 07:45:35.937 2340 ERROR nova.api.openstack.extensions
2018-01-31 07:45:35.938 2340 INFO nova.api.openstack.wsgi
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] HTTP exception thrown: Unexpected API Error.
Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API
log if possible.
<type 'exceptions.TypeError'>
2018-01-31 07:45:35.938 2340 DEBUG nova.api.openstack.wsgi
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] Returning 500 to user: Unexpected API Error.
Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API
log if possible.
<type 'exceptions.TypeError'> __call__
/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1029
2018-01-31 07:45:35.939 2340 INFO nova.osapi_compute.wsgi.server
[req-254ac685-f218-4a23-8fa6-9c6a2d48bb07 45bd2d15e8534c469bf08b7db268e8d4
f80e111e5030457a872bee3a4c11ca70 - 8433db4810f947168950770f8c93a4f2
8433db4810f947168950770f8c93a4f2] 192.168.0.180 "GET
/v2.1/servers/detail?all_tenants=True HTTP/1.1" status: 500 len: 626 time:
0.2633660
but at same tenant I can show a instance information .
nova --os-project-name="Intercon" show 5249185f-0a8e-4b24-a777-214ed967695f
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value
|
+--------------------------------------+----------------------------------------------------------------------------------+
| INTERNAL-NETWORK network | 192.168.120.14
|
| OS-DCF:diskConfig | AUTO
|
| OS-EXT-AZ:availability_zone | XXXXXXXXXXX
|
| OS-EXT-STS:power_state | 1
|
| OS-EXT-STS:task_state | -
|
| OS-EXT-STS:vm_state | active
|
| OS-SRV-USG:launched_at | 2017-10-19T15:30:57.000000
|
| OS-SRV-USG:terminated_at | -
|
| accessIPv4 |
|
| accessIPv6 |
|
| config_drive |
|
| created | 2017-10-19T15:30:47Z
|
| description | zabbix-intercon
|
| flavor:disk | 40
|
| flavor:ephemeral | 0
|
| flavor:extra_specs | {}
|
| flavor:original_name | m1.medium
|
| flavor:ram | 4096
|
| flavor:swap | 0
|
| flavor:vcpus | 2
|
| hostId |
723f077d53d4182fcb76bb87c7fc16efd7ad5fea5d939a9d1a61cb07
|
| id | 5249185f-0a8e-4b24-a777-214ed967695f
|
| image | Centos-7
(d3f18d34-bb00-44e1-85cd-71bfae7ca24a) |
| key_name | saulo-fcm360_id_rsa
|
| locked | False
|
| metadata | {}
|
| name | zabbix-intercon
|
| os-extended-volumes:volumes_attached | [{"id":
"4a30e8a4-86c3-4cef-9f89-5bccefd9df22", "delete_on_termination": false}] |
| progress | 0
|
| security_groups | Zabbix Access, default
|
| status | ACTIVE
|
| tags | []
|
| tenant_id | e082484fc4004da78407a0c0857ae8a9
|
| updated | 2018-01-31T00:06:36Z
|
| user_id | ff9a3af4473c480cacd0c4e18d68edb4
|
+--------------------------------------+----------------------------------------------------------------------------------+
Ant help ?
Saulo Augusto Silva
sauloaugustosilva at gmail.com
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1746509/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
