Public bug reported: nova.tests.functional.libvirt.test_pci_sriov_servers.VDPAServersTest.test_suspend_and_resume failed for one of my patches. Failure itself looks sporadic, but may be a sign of underlying problems.
Complete log: https://3597fc80eb5e70afafd6-8fe1ce567603c46a794844f7885416e9.ssl.cf1.rackcdn.com/873388/2/check/nova- tox-functional-py310/6921a72/job-output.txt My understanding: - "Original exception" [1] is logged when nova-conductor is started - This exception triggers calls, which in turn cause libvirt import and trigger ImportError exception: 2023-02-17 11:43:02.536761 | ubuntu-jammy | 2023-02-17 11:26:59,887 WARNING [nova.tests.fixtures.nova] !!!---!!! TestCase ID nova.tests.functional.libvirt.test_pci_sriov_servers.VDPAServersTest.test_suspend_and_resume hit the import poison while importing libvirt. If you see this in a failed functional test then please let #openstack-nova on IRC know about it. !!!---!!! It looks like original issue is caused by some RPC call related to live migration. This problem seem to be sporadic and is not a blocker, but it looks quite interesting and could affect CI in general, so I am reporting this bug. [1] 2023-02-17 11:43:02.529191 | ubuntu-jammy | 2023-02-17 11:26:59,782 ERROR [root] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 207, in _send\n reply, failure = reply_q.get(timeout=timeout)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get\n return waiter.wait()\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait\n return get_hub().switch()\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\n', '_queue.Empty\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Tra ceback (most recent call last):\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 203, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 9306, in _post_live_migration\n self._update_scheduler_instance_info(ctxt, instance)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 2225, in _update_scheduler_instance_info\n self.query_client.update_instance_info(context, self.host,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 69, in update_instance_info\n self.scheduler_rpcapi.update_instance_info(context, host_name,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/rpcapi.py", line 174, in update_instance_info\n cctxt.cast(ctxt, \'update_instance_info\', host_name=host_name,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/ functional-py310/lib/python3.10/site-packages/fixtures/_fixtures/monkeypatch.py", line 86, in avoid_get\n return captured_method(*args, **kwargs)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in c 2023-02-17 11:43:02.529623 | ubuntu-jammy | all\n result = self.transport._send(\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send\n return self._driver.send(target, ctxt, message,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 213, in _send\n raise oslo_messaging.MessagingTimeout(\n', 'oslo_messaging.exceptions.MessagingTimeout: No reply on topic scheduler\n'] 2023-02-17 11:43:02.529678 | ubuntu-jammy | 2023-02-17 11:26:59,881 WARNING [nova.virt.libvirt.driver] Error monitoring migration: (sqlite3.OperationalError) no such table: compute_nodes ** Affects: nova Importance: Undecided Status: New ** Tags: testing -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2007860 Title: master nova-tox-functional-py310 fails because of VDPAServersTest.test_suspend_and_resume Status in OpenStack Compute (nova): New Bug description: nova.tests.functional.libvirt.test_pci_sriov_servers.VDPAServersTest.test_suspend_and_resume failed for one of my patches. Failure itself looks sporadic, but may be a sign of underlying problems. Complete log: https://3597fc80eb5e70afafd6-8fe1ce567603c46a794844f7885416e9.ssl.cf1.rackcdn.com/873388/2/check/nova- tox-functional-py310/6921a72/job-output.txt My understanding: - "Original exception" [1] is logged when nova-conductor is started - This exception triggers calls, which in turn cause libvirt import and trigger ImportError exception: 2023-02-17 11:43:02.536761 | ubuntu-jammy | 2023-02-17 11:26:59,887 WARNING [nova.tests.fixtures.nova] !!!---!!! TestCase ID nova.tests.functional.libvirt.test_pci_sriov_servers.VDPAServersTest.test_suspend_and_resume hit the import poison while importing libvirt. If you see this in a failed functional test then please let #openstack-nova on IRC know about it. !!!---!!! It looks like original issue is caused by some RPC call related to live migration. This problem seem to be sporadic and is not a blocker, but it looks quite interesting and could affect CI in general, so I am reporting this bug. [1] 2023-02-17 11:43:02.529191 | ubuntu-jammy | 2023-02-17 11:26:59,782 ERROR [root] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 207, in _send\n reply, failure = reply_q.get(timeout=timeout)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/queue.py", line 322, in get\n return waiter.wait()\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/queue.py", line 141, in wait\n return get_hub().switch()\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\n', '_queue.Empty\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'T raceback (most recent call last):\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 203, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 9306, in _post_live_migration\n self._update_scheduler_instance_info(ctxt, instance)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 2225, in _update_scheduler_instance_info\n self.query_client.update_instance_info(context, self.host,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/client/query.py", line 69, in update_instance_info\n self.scheduler_rpcapi.update_instance_info(context, host_name,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/nova/scheduler/rpcapi.py", line 174, in update_instance_info\n cctxt.cast(ctxt, \'update_instance_info\', host_name=host_name,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.to x/functional-py310/lib/python3.10/site-packages/fixtures/_fixtures/monkeypatch.py", line 86, in avoid_get\n return captured_method(*args, **kwargs)\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/rpc/client.py", line 190, in c 2023-02-17 11:43:02.529623 | ubuntu-jammy | all\n result = self.transport._send(\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/transport.py", line 123, in _send\n return self._driver.send(target, ctxt, message,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 222, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout,\n', ' File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py310/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_fake.py", line 213, in _send\n raise oslo_messaging.MessagingTimeout(\n', 'oslo_messaging.exceptions.MessagingTimeout: No reply on topic scheduler\n'] 2023-02-17 11:43:02.529678 | ubuntu-jammy | 2023-02-17 11:26:59,881 WARNING [nova.virt.libvirt.driver] Error monitoring migration: (sqlite3.OperationalError) no such table: compute_nodes To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2007860/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
