Reviewed: https://review.opendev.org/c/openstack/nova/+/826942 Committed: https://opendev.org/openstack/nova/commit/ecb5f1717191379b5ade160ae814f7da8b91e7e8 Submitter: "Zuul (22348)" Branch: master
commit ecb5f1717191379b5ade160ae814f7da8b91e7e8 Author: melanie witt <[email protected]> Date: Fri Jan 28 21:21:06 2022 +0000 Raise InstanceNotFound on fkey constraint fail saving info cache We have seen tracebacks in the gate in the nova-multi-cell job during cross-cell migrations of a failure to save an instance info cache due to a foreign key constraint violation on instance_uuid. This can occur if we happen to receive an external event from neutron after the instance record has been moved to the destination cell. This adds a catch of DBReferenceError from oslo.db and raises InstanceNotFound if the key is instance_uuid, else it reraises. One more step is required to prevent tracing which is to log at level INFO instead of level ERROR in the case of InstanceNotFound. Then reraise it for callers to handle. Closes-Bug: #1944787 Change-Id: I71b7312879dab6857255c78c21ad8dd45d5ea00f ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1944787 Title: Failed storing info cache: ... DBReferenceError (pymysql.err.IntegrityError) ... foreign key constraint fails (`nova_cell1`.`instance_info_caches` ... Status in OpenStack Compute (nova): Fix Released Bug description: Noticed in gate runs in the nova-multi-cell job, tracebacks like the following during cross cell resize: Sep 23 06:10:18.393560 ubuntu-focal-rax-iad-0026631025 nova- compute[110797]: DEBUG nova.compute.manager [None req-782bdf2c-3bee-47ab-8e75-d9c5438638f0 tempest- ServerActionsTestJSON-1400567331 tempest- ServerActionsTestJSON-1400567331-project] [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] Checking if we can cross-cell migrate instance to this host (ubuntu-focal-rax-iad-0026631025). {{(pid=110797) prep_snapshot_based_resize_at_dest /opt/stack/nova/nova/compute/manager.py:5397}} [...] Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [req-da56c934-c427-4c21-bfa5-b26da09b4687 req-08c8d69d-3330-4a23-9ce0-f399d9f61ab0 service nova] [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] Failed storing info cache: oslo_messaging.rpc.client.RemoteError: Remote error: DBReferenceError (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))') Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: [SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)] Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: [parameters: {'created_at': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), 'updated_at': None, 'deleted_at': None, 'deleted': 0, 'network_info': '[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]', 'instance_uuid': 'e9254211-66b3-422d-a486-9c7dc92d0d2f'}] Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n result = self._query(query)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n conn.query(q)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n result.read()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n raise errorclass(errno, errval)\n', "pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')\n", '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/conductor/manager.py", line 139, in _object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper\n return fn(self, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/objects/instance_info_cache.py", line 78, in save\n rv = db.instance_info_cache_update(self._context,\n', ' File "/opt/stack/nova/nova/db/utils.py", line 35, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/db/main/api.py", line 190, in wrapper\n return f(context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/db/main/api.py", line 2447, in instance_info_cache_update\n info_cache.save(context.session)\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save\n session.flush()\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3298, in flush\n self._flush(objects)\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3438, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n raise exception\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3398, in _flush\n flush_context.execute()\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 242, in save_obj\n _emit_insert_statements(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1219, in _emit_insert_statements\n result = connection._execute_20(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1452, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1814, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1993, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n raise exception\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n result = self._query(query)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n conn.query(q)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n result.read()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, \'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))\')\n[SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]\n[parameters: {\'created_at\': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), \'updated_at\': None, \'deleted_at\': None, \'deleted\': 0, \'network_info\': \'[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]\', \'instance_uuid\': \'e9254211-66b3-422d-a486-9c7dc92d0d2f\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n']. Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] Traceback (most recent call last): Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] File "/opt/stack/nova/nova/network/neutron.py", line 125, in update_instance_cache_with_nw_info Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] ic.save() Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 209, in wrapper Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] updates, result = self.indirection_api.object_action( Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] File "/opt/stack/nova/nova/conductor/rpcapi.py", line 247, in object_action Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] return cctxt.call(context, 'object_action', objinst=objinst, Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/client.py", line 189, in call Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] result = self.transport._send( Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/transport.py", line 123, in _send Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] return self._driver.send(target, ctxt, message, Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 680, in send Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] return self._send(target, ctxt, message, wait_for_reply, timeout, Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 672, in _send Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] raise result Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] oslo_messaging.rpc.client.RemoteError: Remote error: DBReferenceError (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))') Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] [SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)] Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] [parameters: {'created_at': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), 'updated_at': None, 'deleted_at': None, 'deleted': 0, 'network_info': '[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]', 'instance_uuid': 'e9254211-66b3-422d-a486-9c7dc92d0d2f'}] Sep 23 06:10:45.745949 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 23 06:10:45.747562 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n result = self._query(query)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n conn.query(q)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n result.read()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n raise errorclass(errno, errval)\n', "pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))')\n", '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/conductor/manager.py", line 139, in _object_dispatch\n return getattr(target, method)(*args, **kwargs)\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper\n return fn(self, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/objects/instance_info_cache.py", line 78, in save\n rv = db.instance_info_cache_update(self._context,\n', ' File "/opt/stack/nova/nova/db/utils.py", line 35, in wrapper\n return f(*args, **kwargs)\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper\n ectxt.value = e.inner_exc\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__\n self.force_reraise()\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise\n raise self.value\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper\n return f(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/db/main/api.py", line 190, in wrapper\n return f(context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/db/main/api.py", line 2447, in instance_info_cache_update\n info_cache.save(context.session)\n', ' File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save\n session.flush()\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3298, in flush\n self._flush(objects)\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3438, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__\n compat.raise_(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n raise exception\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3398, in _flush\n flush_context.execute()\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute\n rec.execute(self)\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute\n util.preloaded.orm_persistence.save_obj(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 242, in save_obj\n _emit_insert_statements(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1219, in _emit_insert_statements\n result = connection._execute_20(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in _execute_20\n return meth(self, args_10style, kwargs_10style, execution_options)\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection\n return connection._execute_clauseelement(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1452, in _execute_clauseelement\n ret = self._execute_context(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1814, in _execute_context\n self._handle_dbapi_exception(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1993, in _handle_dbapi_exception\n util.raise_(newraise, with_traceback=exc_info[2], from_=e)\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_\n raise exception\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1771, in _execute_context\n self.dialect.do_execute(\n', ' File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 717, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute\n result = self._query(query)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query\n conn.query(q)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result\n result.read()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet\n packet.raise_for_error()\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception\n raise errorclass(errno, errval)\n', 'oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, \'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` FOREIGN KEY (`instance_uuid`) REFERENCES `instances` (`uuid`))\')\n[SQL: INSERT INTO instance_info_caches (created_at, updated_at, deleted_at, deleted, network_info, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(network_info)s, %(instance_uuid)s)]\n[parameters: {\'created_at\': datetime.datetime(2021, 9, 23, 6, 10, 45, 736541), \'updated_at\': None, \'deleted_at\': None, \'deleted\': 0, \'network_info\': \'[{"id": "6cda3ae0-0087-4c9d-9aa4-307aaac11229", "address": "fa:16:3e:42:88:b5", "network": {"id": "15d8321e-4160-46c0-aacc-fdfa0e57061e", "bridge": " ... (720 characters truncated) ... : null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "delegate_create": true, "meta": {}}]\', \'instance_uuid\': \'e9254211-66b3-422d-a486-9c7dc92d0d2f\'}]\n(Background on this error at: https://sqlalche.me/e/14/gkpj)\n']. Sep 23 06:10:45.747562 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: ERROR nova.network.neutron [instance: e9254211-66b3-422d-a486-9c7dc92d0d2f] A cursory look through the code it looks like this is an external neutron event coming in after the instance has moved to another cell, so it is not found in the expected cell that was looked up initially when the external event was received. This doesn't necessarily cause a job failure, so it appears mostly be a traceback we need to clean up. Maybe we could DEBUG or INFO log that we possibly received an event for an instance that has moved across cells. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1944787/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
