Reviewed: https://review.opendev.org/c/openstack/nova/+/812392 Committed: https://opendev.org/openstack/nova/commit/cf4e516f553ccbc4fad6014153d9b51095297607 Submitter: "Zuul (22348)" Branch: master
commit cf4e516f553ccbc4fad6014153d9b51095297607 Author: Lee Yarwood <[email protected]> Date: Mon Oct 4 18:11:39 2021 +0100 zuul: Move live migration jobs back to voting With the resolution of bug #1945983 within devstack we can now move our live migration jobs back to voting. Related-Bug: #1945983 Closes-Bug: #1912310 Depends-On: https://review.opendev.org/c/openstack/devstack/+/812391 Depends-On: https://review.opendev.org/c/openstack/devstack/+/812925 Change-Id: I25177554802579952510c73985287fd76681012c ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1912310 Title: libvirt.libvirtError: unable to connect to server at Status in OpenStack Compute (nova): Fix Released Bug description: Description =========== nova-live-migration fails due to the following error on the src during a migration test: https://zuul.opendev.org/t/openstack/build/76c37d93ba274cd6b90fbe0659255959/log/compute1/logs/screen- n-cpu.txt Jan 18 12:03:25.145915 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: ERROR nova.virt.libvirt.driver [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Migration operation has aborted Jan 18 12:03:25.194466 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: INFO nova.compute.manager [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Swapping old allocation on dict_keys(['022fd821-f027-4019-ac4b-cb1e1d2d5423']) held by migration 7b332f03-b32e-4cc8-8e2d-23ed7bee03bd for instance Jan 18 12:03:25.239903 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: ERROR nova.virt.libvirt.driver [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Live Migration failure: unable to connect to server at 'ubuntu-focal-limestone-regionone-0022589715:49152': Connection refused: libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-limestone-regionone-0022589715:49152': Connection refused Jan 18 12:03:25.240119 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: DEBUG nova.virt.libvirt.driver [-] [instance: 9b9d2344-6c8e-4dba-a8f9-60f86e52c854] Migration operation thread notification {{(pid=52916) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:9448}} Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: Traceback (most recent call last): Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: timer() Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/timer.py", line 59, in __call__ Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: cb(*args, **kw) Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/eventlet/event.py", line 175, in _do_send Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: waiter.switch(result) Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/eventlet/greenthread.py", line 221, in main Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: result = function(*args, **kwargs) Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/opt/stack/nova/nova/utils.py", line 660, in context_wrapper Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: return func(*args, **kwargs) Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 9103, in _live_migration_operation Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: LOG.error("Live Migration failure: %s", e, instance=instance) Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: self.force_reraise() Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: six.reraise(self.type_, self.value, self.tb) Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: raise value Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 9091, in _live_migration_operation Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: guest.migrate(self._live_migration_uri(dest), Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 649, in migrate Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: self._domain.migrateToURI3( Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 190, in doit Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: result = proxy_call(self._autowrap, f, *args, **kwargs) Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 148, in proxy_call Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: rv = execute(f, *args, **kwargs) Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 129, in execute Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: six.reraise(c, e, tb) Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: raise value Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 83, in tworker Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: rv = meth(*args, **kwargs) Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: File "/usr/local/lib/python3.8/dist-packages/libvirt.py", line 2099, in migrateToURI3 Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: raise libvirtError('virDomainMigrateToURI3() failed') Jan 18 12:03:25.251131 ubuntu-focal-limestone-regionone-0022589731 nova-compute[52916]: libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-limestone-regionone-0022589715:49152': Connection refused Steps to reproduce ================== $ cat queries/libvirt.yaml query: >- message:"libvirt.libvirtError: unable to connect to server at" AND tags:screen-n-cpu.txt $ elastic-recheck-query queries/libvirt.yaml total hits: 12 build_branch 83% master 16% stable/ussuri build_change 16% 711625 16% 743231 16% 754702 16% 764401 16% 770547 build_hostids 16% 13cc9efd4550bb9c731b89fae9c27cc77058360313dea9eebe140cac 1ccf2ea621af4ba5d658324f8716441c624749f7847700ac6aa69490 16% 322bfced1bb62c11480e9c8cf97e066ab7cb1067b23b38e8004c6f69 23320365a917d59498b30bb8308684708801795984a6ca42b1cd06b7 16% 422fc847b99ec49a4a48de150fd4c453e34d4b85847603a3435e1511 e1ef7b86aaeb7f61d567417ced5d641b20cff65c4c36e91a6173705f 16% 77681beb0ddf7a44597e1e8fa6f648861b5edf5445510df6083cdb72 c4902db85fd9c45a1c8b61c8e612d117a8a8f2393128fd1cae285326 16% 82ed50455b38e6ed7725b8020a977c59f2aaa9db51f018ed12e648b2 3a893be3caac821988438433df3cb21c389ffed93878670819a924f4 24505d497fe40296a37e97a5a1aa1caf33a2c685b883fb60eabf95d5 build_name 16% devstack-plugin-ceph-multinode-tempest-py3 16% neutron-ovn-tempest-full-multinode-ovs-master 16% neutron-tempest-dvr-ha-multinode-full 16% nova-live-migration 16% nova-next build_node 83% ubuntu-focal 16% ubuntu-bionic build_queue 100% check build_status 100% FAILURE build_zuul_url 100% N/A filename 100% compute1/logs/screen-n-cpu.txt log_url 16% https://7b4153b100b6f570d4fc-1ca4271300c9fb8c2811a73921fc79fe.ssl.cf2.rackcdn.com/764401/14/check/neutron-tempest-dvr-ha-multinode-full/d253afd/compute1/logs/screen-n-cpu.txt 16% https://b0f3338313a78e19953a-f09976335bb1c2233a92ccf49f573637.ssl.cf5.rackcdn.com/754702/5/check/nova-live-migration/76c37d9/compute1/logs/screen-n-cpu.txt 16% https://cfb2c6e30c49a7b551c1-e4b92f5e1fb788c5da82bf1447ffa549.ssl.cf2.rackcdn.com/711625/27/check/devstack-plugin-ceph-multinode-tempest-py3/677c92c/compute1/logs/screen-n-cpu.txt 16% https://e7591f9adbfbe7dc24c4-cd07d3426af01024dbeaac2f8639d2e0.ssl.cf2.rackcdn.com/743231/9/check/nova-next/cbe27d5/compute1/logs/screen-n-cpu.txt 16% https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_09a/770547/1/check/tempest-multinode-full-py3/09a5f18/compute1/logs/screen-n-cpu.txt loglevel 100% ERROR module 100% nova.virt.libvirt.driver node_provider 33% ovh-bhs1 16% airship-kna1 16% limestone-regionone 16% rax-ord 16% vexxhost-ca-ymq-1 port 16% 39628 16% 41358 16% 46144 16% 52362 16% 58212 project 50% openstack/neutron 33% openstack/nova 16% openstack/devstack-plugin-ceph syslog_pid 16% 19491 16% 46189 16% 46724 16% 52916 16% 53510 syslog_program 16% ubuntu-bionic-ovh-bhs1-0022571568 nova-compute 16% ubuntu-focal-airship-kna1-0022565736 nova-compute 16% ubuntu-focal-limestone-regionone-0022589731 nova-compute 16% ubuntu-focal-ovh-bhs1-0022599932 nova-compute 16% ubuntu-focal-rax-ord-0022553677 nova-compute tags 50% screen-n-cpu.txt screen oslofmt 50% screen-n-cpu.txt screen oslofmt _grokparsefailure voting 50% 0 50% 1 zuul_attempts 83% 1 16% 2 zuul_executor 33% ze06.openstack.org 33% ze11.openstack.org 16% ze04.openstack.org 16% ze09.openstack.org Expected result =============== No connection issues with libvirtd. Actual result ============= Connection issues with libvirtd cause CI runs to fail. Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ As above, master and stable/ussuri appear impacted thus far. 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? libvirt 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? N/A 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs ============== To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1912310/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
