Public bug reported:
Noticed in gate runs in the nova-multi-cell job, tracebacks like the
following during cross cell resize:
Sep 23 06:10:18.393560 ubuntu-focal-rax-iad-0026631025 nova-
compute[110797]: DEBUG nova.compute.manager [None
req-782bdf2c-3bee-47ab-8e75-d9c5438638f0 tempest-
ServerActionsTestJSON-1400567331 tempest-
ServerActionsTestJSON-1400567331-project] [instance:
e9254211-66b3-422d-a486-9c7dc92d0d2f] Checking if we can cross-cell
migrate instance to this host (ubuntu-focal-rax-iad-0026631025).
{{(pid=110797) prep_snapshot_based_resize_at_dest
/opt/stack/nova/nova/compute/manager.py:5397}}
[...]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [req-da56c934-c427-4c21-bfa5-b26da09b4687
req-08c8d69d-3330-4a23-9ce0-f399d9f61ab0 service nova] [instance:
e9254211-66b3-422d-a486-9c7dc92d0d2f] Failed storing info cache:
oslo_messaging.rpc.client.RemoteError: Remote error: DBReferenceError
(pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a
foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))')
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
[SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at,
deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s,
%(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
[parameters: {'created_at': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541),
'updated_at': None, 'deleted_at': None, 'deleted': 0, 'network_info': '[{"id":
"6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5",
"network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720
characters truncated) ... : null, "qbg_params": null, "active": false,
"vnic_type": "normal", "profile": {}, "preserve_on_delete": false,
"delegate_create": true, "meta": {}}]', 'instance_uuid':
'e9254211-66b3-422d-a486-9c7dc92d0d2f'}]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
(Background on this error at: https://sqlalche.me/e/14/gkpj)
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
['Traceback (most recent call last):\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771,
in _execute_context\n self.dialect.do_execute(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line
717, in do_execute\n cursor.execute(statement, parameters)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in
execute\n result = self._query(query)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in
_query\n conn.query(q)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in
query\n self._affected_rows =
self._read_query_result(unbuffered=unbuffered)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in
_read_query_result\n result.read()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in
read\n first_packet = self.connection._read_packet()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in
_read_packet\n packet.raise_for_error()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in
raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in
raise_mysql_exception\n raise errorclass(errno, errval)\n',
"pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a
foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))')\n", '\nThe above exception was the direct
cause of the following exception:\n\n', 'Traceback (most recent call last):\n',
' File "/opt/stack/nova/nova/conductor/manager.py", line 139, in
_object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', '
File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py",
line 226, in wrapper\n return fn(self, *args, **kwargs)\n', ' File
"/opt/stack/nova/nova/objects/instance_info_cache.py", line 78, in save\n rv
= db.instance_info_cache_update(self._context,\n', ' File
"/opt/stack/nova/nova/db/utils.py", line 35, in wrapper\n return f(*args,
**kwargs)\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py",
line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in
__exit__\n self.force_reraise()\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in
force_reraise\n raise self.value\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper\n
return f(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/db/main/api.py",
line 190, in wrapper\n return f(context, *args, **kwargs)\n', ' File
"/opt/stack/nova/nova/db/main/api.py", line 2447, in
instance_info_cache_update\n info_cache.save(context.session)\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/models.py", line 48,
in save\n session.flush()\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3298,
in flush\n self._flush(objects)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3438,
in _flush\n transaction.rollback(_capture_exception=True)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line
70, in __exit__\n compat.raise_(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207,
in raise_\n raise exception\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3398,
in _flush\n flush_context.execute()\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line
456, in execute\n rec.execute(self)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line
630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line
242, in save_obj\n _emit_insert_statements(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line
1219, in _emit_insert_statements\n result = connection._execute_20(\n', '
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line
1583, in _execute_20\n return meth(self, args_10style, kwargs_10style,
execution_options)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 323,
in _execute_on_connection\n return connection._execute_clauseelement(\n', '
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line
1452, in _execute_clauseelement\n ret = self._execute_context(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1814,
in _execute_context\n self._handle_dbapi_exception(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1993,
in _handle_dbapi_exception\n util.raise_(newraise,
with_traceback=exc_info[2], from_=e)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207,
in raise_\n raise exception\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771,
in _execute_context\n self.dialect.do_execute(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line
717, in do_execute\n cursor.execute(statement, parameters)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in
execute\n result = self._query(query)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in
_query\n conn.query(q)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in
query\n self._affected_rows =
self._read_query_result(unbuffered=unbuffered)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in
_read_query_result\n result.read()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in
read\n first_packet = self.connection._read_packet()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in
_read_packet\n packet.raise_for_error()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in
raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in
raise_mysql_exception\n raise errorclass(errno, errval)\n',
'oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452,
\'Cannot add or update a child row: a foreign key constraint fails
(`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))\')\n[SQL: INSERT INTO instance_info_caches
(created_at, updated_at, deleted_at, deleted, network_info, instance_uuid)
VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s,
%(network_info)s, %(instance_uuid)s)]\n[parameters: {\'created_at\':
datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), \'updated_at\': None,
\'deleted_at\': None, \'deleted\': 0, \'network_info\': \'[{"id":
"6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5",
"network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720
characters truncated) ... : null, "qbg_params": null, "active": false,
"vnic_type": "normal", "profile": {}, "preserve_on_delete": false,
"delegate_create": true, "meta": {}}]\', \'instance_uuid\':
\'e9254211-66b3-422d-a486-9c7dc92d0d2f\'}]\n(Background on this error at:
https://sqlalche.me/e/14/gkpj)\n'].
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
Traceback (most recent call last):
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File "/opt/stack/nova/nova/network/neutron.py", line 125, in
update_instance_cache_with_nw_info
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
ic.save()
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py",
line 209, in wrapper
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
updates, result = self.indirection_api.object_action(
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File "/opt/stack/nova/nova/conductor/rpcapi.py", line 247, in object_action
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
return cctxt.call(context, 'object_action', objinst=objinst,
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/client.py",
line 189, in call
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
result = self.transport._send(
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/transport.py", line
123, in _send
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
return self._driver.send(target, ctxt, message,
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 680, in send
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
return self._send(target, ctxt, message, wait_for_reply, timeout,
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 672, in _send
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
raise result
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
oslo_messaging.rpc.client.RemoteError: Remote error: DBReferenceError
(pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a
foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))')
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
[SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at,
deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s,
%(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
[parameters: {'created_at': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541),
'updated_at': None, 'deleted_at': None, 'deleted': 0, 'network_info': '[{"id":
"6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5",
"network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720
characters truncated) ... : null, "qbg_params": null, "active": false,
"vnic_type": "normal", "profile": {}, "preserve_on_delete": false,
"delegate_create": true, "meta": {}}]', 'instance_uuid':
'e9254211-66b3-422d-a486-9c7dc92d0d2f'}]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
Sep 23 06:10:45.747562 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
['Traceback (most recent call last):\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771,
in _execute_context\n self.dialect.do_execute(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line
717, in do_execute\n cursor.execute(statement, parameters)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in
execute\n result = self._query(query)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in
_query\n conn.query(q)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in
query\n self._affected_rows =
self._read_query_result(unbuffered=unbuffered)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in
_read_query_result\n result.read()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in
read\n first_packet = self.connection._read_packet()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in
_read_packet\n packet.raise_for_error()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in
raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in
raise_mysql_exception\n raise errorclass(errno, errval)\n',
"pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a
foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))')\n", '\nThe above exception was the direct
cause of the following exception:\n\n', 'Traceback (most recent call last):\n',
' File "/opt/stack/nova/nova/conductor/manager.py", line 139, in
_object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', '
File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py",
line 226, in wrapper\n return fn(self, *args, **kwargs)\n', ' File
"/opt/stack/nova/nova/objects/instance_info_cache.py", line 78, in save\n rv
= db.instance_info_cache_update(self._context,\n', ' File
"/opt/stack/nova/nova/db/utils.py", line 35, in wrapper\n return f(*args,
**kwargs)\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py",
line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in
__exit__\n self.force_reraise()\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in
force_reraise\n raise self.value\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper\n
return f(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/db/main/api.py",
line 190, in wrapper\n return f(context, *args, **kwargs)\n', ' File
"/opt/stack/nova/nova/db/main/api.py", line 2447, in
instance_info_cache_update\n info_cache.save(context.session)\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/models.py", line 48,
in save\n session.flush()\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3298,
in flush\n self._flush(objects)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3438,
in _flush\n transaction.rollback(_capture_exception=True)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line
70, in __exit__\n compat.raise_(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207,
in raise_\n raise exception\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3398,
in _flush\n flush_context.execute()\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line
456, in execute\n rec.execute(self)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line
630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line
242, in save_obj\n _emit_insert_statements(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line
1219, in _emit_insert_statements\n result = connection._execute_20(\n', '
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line
1583, in _execute_20\n return meth(self, args_10style, kwargs_10style,
execution_options)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 323,
in _execute_on_connection\n return connection._execute_clauseelement(\n', '
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line
1452, in _execute_clauseelement\n ret = self._execute_context(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1814,
in _execute_context\n self._handle_dbapi_exception(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1993,
in _handle_dbapi_exception\n util.raise_(newraise,
with_traceback=exc_info[2], from_=e)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207,
in raise_\n raise exception\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771,
in _execute_context\n self.dialect.do_execute(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line
717, in do_execute\n cursor.execute(statement, parameters)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in
execute\n result = self._query(query)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in
_query\n conn.query(q)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in
query\n self._affected_rows =
self._read_query_result(unbuffered=unbuffered)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in
_read_query_result\n result.read()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in
read\n first_packet = self.connection._read_packet()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in
_read_packet\n packet.raise_for_error()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in
raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in
raise_mysql_exception\n raise errorclass(errno, errval)\n',
'oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452,
\'Cannot add or update a child row: a foreign key constraint fails
(`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))\')\n[SQL: INSERT INTO instance_info_caches
(created_at, updated_at, deleted_at, deleted, network_info, instance_uuid)
VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s,
%(network_info)s, %(instance_uuid)s)]\n[parameters: {\'created_at\':
datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), \'updated_at\': None,
\'deleted_at\': None, \'deleted\': 0, \'network_info\': \'[{"id":
"6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5",
"network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720
characters truncated) ... : null, "qbg_params": null, "active": false,
"vnic_type": "normal", "profile": {}, "preserve_on_delete": false,
"delegate_create": true, "meta": {}}]\', \'instance_uuid\':
\'e9254211-66b3-422d-a486-9c7dc92d0d2f\'}]\n(Background on this error at:
https://sqlalche.me/e/14/gkpj)\n'].
Sep 23 06:10:45.747562 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
A cursory look through the code it looks like this is an external neutron event
coming in after the instance has moved to another cell, so it is not found in
the expected cell that was looked up initially when the external event was
received.
This doesn't necessarily cause a job failure, so it appears mostly be a
traceback we need to clean up. Maybe we could DEBUG or INFO log that we
possibly received an event for an instance that has moved across cells.
** Affects: nova
Importance: Undecided
Status: New
** Tags: cells
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1944787
Title:
Failed storing info cache: ... DBReferenceError
(pymysql.err.IntegrityError) ... foreign key constraint fails
(`nova_cell1`.`instance_info_caches` ...
Status in OpenStack Compute (nova):
New
Bug description:
Noticed in gate runs in the nova-multi-cell job, tracebacks like the
following during cross cell resize:
Sep 23 06:10:18.393560 ubuntu-focal-rax-iad-0026631025 nova-
compute[110797]: DEBUG nova.compute.manager [None
req-782bdf2c-3bee-47ab-8e75-d9c5438638f0 tempest-
ServerActionsTestJSON-1400567331 tempest-
ServerActionsTestJSON-1400567331-project] [instance:
e9254211-66b3-422d-a486-9c7dc92d0d2f] Checking if we can cross-cell
migrate instance to this host (ubuntu-focal-rax-iad-0026631025).
{{(pid=110797) prep_snapshot_based_resize_at_dest
/opt/stack/nova/nova/compute/manager.py:5397}}
[...]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [req-da56c934-c427-4c21-bfa5-b26da09b4687
req-08c8d69d-3330-4a23-9ce0-f399d9f61ab0 service nova] [instance:
e9254211-66b3-422d-a486-9c7dc92d0d2f] Failed storing info cache:
oslo_messaging.rpc.client.RemoteError: Remote error: DBReferenceError
(pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a
foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))')
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
[SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at,
deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s,
%(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
[parameters: {'created_at': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541),
'updated_at': None, 'deleted_at': None, 'deleted': 0, 'network_info': '[{"id":
"6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5",
"network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720
characters truncated) ... : null, "qbg_params": null, "active": false,
"vnic_type": "normal", "profile": {}, "preserve_on_delete": false,
"delegate_create": true, "meta": {}}]', 'instance_uuid':
'e9254211-66b3-422d-a486-9c7dc92d0d2f'}]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
(Background on this error at: https://sqlalche.me/e/14/gkpj)
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
['Traceback (most recent call last):\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771,
in _execute_context\n self.dialect.do_execute(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line
717, in do_execute\n cursor.execute(statement, parameters)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in
execute\n result = self._query(query)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in
_query\n conn.query(q)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in
query\n self._affected_rows =
self._read_query_result(unbuffered=unbuffered)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in
_read_query_result\n result.read()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in
read\n first_packet = self.connection._read_packet()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in
_read_packet\n packet.raise_for_error()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in
raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in
raise_mysql_exception\n raise errorclass(errno, errval)\n',
"pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a
foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))')\n", '\nThe above exception was the direct
cause of the following exception:\n\n', 'Traceback (most recent call last):\n',
' File "/opt/stack/nova/nova/conductor/manager.py", line 139, in
_object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', '
File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py",
line 226, in wrapper\n return fn(self, *args, **kwargs)\n', ' File
"/opt/stack/nova/nova/objects/instance_info_cache.py", line 78, in save\n rv
= db.instance_info_cache_update(self._context,\n', ' File
"/opt/stack/nova/nova/db/utils.py", line 35, in wrapper\n return f(*args,
**kwargs)\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py",
line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in
__exit__\n self.force_reraise()\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in
force_reraise\n raise self.value\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper\n
return f(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/db/main/api.py",
line 190, in wrapper\n return f(context, *args, **kwargs)\n', ' File
"/opt/stack/nova/nova/db/main/api.py", line 2447, in
instance_info_cache_update\n info_cache.save(context.session)\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/models.py", line 48,
in save\n session.flush()\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3298,
in flush\n self._flush(objects)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3438,
in _flush\n transaction.rollback(_capture_exception=True)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line
70, in __exit__\n compat.raise_(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207,
in raise_\n raise exception\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3398,
in _flush\n flush_context.execute()\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line
456, in execute\n rec.execute(self)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line
630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line
242, in save_obj\n _emit_insert_statements(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line
1219, in _emit_insert_statements\n result = connection._execute_20(\n', '
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line
1583, in _execute_20\n return meth(self, args_10style, kwargs_10style,
execution_options)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 323,
in _execute_on_connection\n return connection._execute_clauseelement(\n', '
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line
1452, in _execute_clauseelement\n ret = self._execute_context(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1814,
in _execute_context\n self._handle_dbapi_exception(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1993,
in _handle_dbapi_exception\n util.raise_(newraise,
with_traceback=exc_info[2], from_=e)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207,
in raise_\n raise exception\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771,
in _execute_context\n self.dialect.do_execute(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line
717, in do_execute\n cursor.execute(statement, parameters)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in
execute\n result = self._query(query)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in
_query\n conn.query(q)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in
query\n self._affected_rows =
self._read_query_result(unbuffered=unbuffered)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in
_read_query_result\n result.read()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in
read\n first_packet = self.connection._read_packet()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in
_read_packet\n packet.raise_for_error()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in
raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in
raise_mysql_exception\n raise errorclass(errno, errval)\n',
'oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452,
\'Cannot add or update a child row: a foreign key constraint fails
(`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))\')\n[SQL: INSERT INTO instance_info_caches
(created_at, updated_at, deleted_at, deleted, network_info, instance_uuid)
VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s,
%(network_info)s, %(instance_uuid)s)]\n[parameters: {\'created_at\':
datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), \'updated_at\': None,
\'deleted_at\': None, \'deleted\': 0, \'network_info\': \'[{"id":
"6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5",
"network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720
characters truncated) ... : null, "qbg_params": null, "active": false,
"vnic_type": "normal", "profile": {}, "preserve_on_delete": false,
"delegate_create": true, "meta": {}}]\', \'instance_uuid\':
\'e9254211-66b3-422d-a486-9c7dc92d0d2f\'}]\n(Background on this error at:
https://sqlalche.me/e/14/gkpj)\n'].
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
Traceback (most recent call last):
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File "/opt/stack/nova/nova/network/neutron.py", line 125, in
update_instance_cache_with_nw_info
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
ic.save()
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py",
line 209, in wrapper
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
updates, result = self.indirection_api.object_action(
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File "/opt/stack/nova/nova/conductor/rpcapi.py", line 247, in object_action
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
return cctxt.call(context, 'object_action', objinst=objinst,
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/client.py",
line 189, in call
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
result = self.transport._send(
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/transport.py", line
123, in _send
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
return self._driver.send(target, ctxt, message,
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 680, in send
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
return self._send(target, ctxt, message, wait_for_reply, timeout,
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
File
"/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 672, in _send
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
raise result
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
oslo_messaging.rpc.client.RemoteError: Remote error: DBReferenceError
(pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a
foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))')
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
[SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at,
deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s,
%(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
[parameters: {'created_at': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541),
'updated_at': None, 'deleted_at': None, 'deleted': 0, 'network_info': '[{"id":
"6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5",
"network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720
characters truncated) ... : null, "qbg_params": null, "active": false,
"vnic_type": "normal", "profile": {}, "preserve_on_delete": false,
"delegate_create": true, "meta": {}}]', 'instance_uuid':
'e9254211-66b3-422d-a486-9c7dc92d0d2f'}]
Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
Sep 23 06:10:45.747562 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
['Traceback (most recent call last):\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771,
in _execute_context\n self.dialect.do_execute(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line
717, in do_execute\n cursor.execute(statement, parameters)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in
execute\n result = self._query(query)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in
_query\n conn.query(q)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in
query\n self._affected_rows =
self._read_query_result(unbuffered=unbuffered)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in
_read_query_result\n result.read()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in
read\n first_packet = self.connection._read_packet()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in
_read_packet\n packet.raise_for_error()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in
raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in
raise_mysql_exception\n raise errorclass(errno, errval)\n',
"pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a
foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))')\n", '\nThe above exception was the direct
cause of the following exception:\n\n', 'Traceback (most recent call last):\n',
' File "/opt/stack/nova/nova/conductor/manager.py", line 139, in
_object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', '
File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py",
line 226, in wrapper\n return fn(self, *args, **kwargs)\n', ' File
"/opt/stack/nova/nova/objects/instance_info_cache.py", line 78, in save\n rv
= db.instance_info_cache_update(self._context,\n', ' File
"/opt/stack/nova/nova/db/utils.py", line 35, in wrapper\n return f(*args,
**kwargs)\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py",
line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in
__exit__\n self.force_reraise()\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in
force_reraise\n raise self.value\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper\n
return f(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/db/main/api.py",
line 190, in wrapper\n return f(context, *args, **kwargs)\n', ' File
"/opt/stack/nova/nova/db/main/api.py", line 2447, in
instance_info_cache_update\n info_cache.save(context.session)\n', ' File
"/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/models.py", line 48,
in save\n session.flush()\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3298,
in flush\n self._flush(objects)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3438,
in _flush\n transaction.rollback(_capture_exception=True)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line
70, in __exit__\n compat.raise_(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207,
in raise_\n raise exception\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3398,
in _flush\n flush_context.execute()\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line
456, in execute\n rec.execute(self)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line
630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line
242, in save_obj\n _emit_insert_statements(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line
1219, in _emit_insert_statements\n result = connection._execute_20(\n', '
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line
1583, in _execute_20\n return meth(self, args_10style, kwargs_10style,
execution_options)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 323,
in _execute_on_connection\n return connection._execute_clauseelement(\n', '
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line
1452, in _execute_clauseelement\n ret = self._execute_context(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1814,
in _execute_context\n self._handle_dbapi_exception(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1993,
in _handle_dbapi_exception\n util.raise_(newraise,
with_traceback=exc_info[2], from_=e)\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207,
in raise_\n raise exception\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771,
in _execute_context\n self.dialect.do_execute(\n', ' File
"/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line
717, in do_execute\n cursor.execute(statement, parameters)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in
execute\n result = self._query(query)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in
_query\n conn.query(q)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in
query\n self._affected_rows =
self._read_query_result(unbuffered=unbuffered)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in
_read_query_result\n result.read()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in
read\n first_packet = self.connection._read_packet()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in
_read_packet\n packet.raise_for_error()\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in
raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File
"/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in
raise_mysql_exception\n raise errorclass(errno, errval)\n',
'oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452,
\'Cannot add or update a child row: a foreign key constraint fails
(`nova_cell1`.`instance_info_caches`, CONSTRAINT
`instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`)
REFERENCES `instances` (`uuid`))\')\n[SQL: INSERT INTO instance_info_caches
(created_at, updated_at, deleted_at, deleted, network_info, instance_uuid)
VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s,
%(network_info)s, %(instance_uuid)s)]\n[parameters: {\'created_at\':
datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), \'updated_at\': None,
\'deleted_at\': None, \'deleted\': 0, \'network_info\': \'[{"id":
"6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5",
"network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720
characters truncated) ... : null, "qbg_params": null, "active": false,
"vnic_type": "normal", "profile": {}, "preserve_on_delete": false,
"delegate_create": true, "meta": {}}]\', \'instance_uuid\':
\'e9254211-66b3-422d-a486-9c7dc92d0d2f\'}]\n(Background on this error at:
https://sqlalche.me/e/14/gkpj)\n'].
Sep 23 06:10:45.747562 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]:
ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f]
A cursory look through the code it looks like this is an external neutron
event coming in after the instance has moved to another cell, so it is not
found in the expected cell that was looked up initially when the external event
was received.
This doesn't necessarily cause a job failure, so it appears mostly be
a traceback we need to clean up. Maybe we could DEBUG or INFO log that
we possibly received an event for an instance that has moved across
cells.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1944787/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
