This is not a big, found solution.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2059962
Title:
Unable to Live Migrate Instance. Horizon doesn't show error, logs do
Status in OpenStack Compute (nova):
Invalid
Bug description:
When trying to live migrate an instance between hosts, I get the error
below. I can migrate an instance that is shutdown.
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.665 27905 DEBUG nova.compute.manager [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] live_migration data is
LibvirtLiveMigrateData(bdms=[LibvirtLiveMigrateBDMInfo],block_migration=False,disk_available_mb=63488,disk_over_commit=False,dst_numa_info=<?>,dst_supports_numa_live_migration=<?>,dst_wants_file_backed_memory=False,file_backed_memory_discard=<?>,filename='tmpe_9d4avw',graphics_listen_addr_spice=127.0.0.1,graphics_listen_addr_vnc=10.136.149.5,image_type='default',instance_relative_path='b44705f4-6813-42ad-a4a8-a64255e2a6b7',is_shared_block_storage=True,is_shared_instance_path=False,is_volume_backed=True,migration=Migration(7e5bb92b-8d84-4320-b379-ee7b14fed53d),old_vol_attachment_ids={375dff72-24d4-475c-a172-33e356765410='1432f021-67a6-4b51-9988-fcad5a6e4684'},serial_listen_addr=None,serial_listen_ports=[],src_supports
_native_luks=<?>,src_supports_numa_live_migration=<?>,supported_perf_events=[],target_connect_addr='10.136.149.5',vifs=[VIFMigrateData],wait_for_vif_plugged=True)
_do_live_migration
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/compute/manager.py:8988
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.668 27905 DEBUG nova.objects.instance [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] Lazy-loading
'migration_context' on Instance uuid b44705f4-6813-42ad-a4a8-a64255e2a6b7
obj_load_attr
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/objects/instance.py:1152
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.668 27905 DEBUG nova.virt.libvirt.driver [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance:
b44705f4-6813-42ad-a4a8-a64255e2a6b7] Starting monitoring of live migration
_live_migration
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py:10641
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.671 27905 DEBUG nova.virt.libvirt.driver [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance:
b44705f4-6813-42ad-a4a8-a64255e2a6b7] Operation thread is still running
_live_migration_monitor
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py:10442
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.671 27905 DEBUG nova.virt.libvirt.driver [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance:
b44705f4-6813-42ad-a4a8-a64255e2a6b7] Migration not running yet
_live_migration_monitor
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py:10451
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.681 27905 DEBUG nova.virt.libvirt.migration [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] Find same serial number:
pos=0, serial=375dff72-24d4-475c-a172-33e356765410 _update_volume_xml
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/migration.py:242
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.682 27905 DEBUG nova.virt.libvirt.vif [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] vif_type=ovs
instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=True,availability_zone='nova',cell_name=None,cleaned=False,compute_id=15,config_drive='',created_at=2024-04-01T16:44:13Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,device_metadata=<?>,disable_terminate=False,display_description=None,display_name='test',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(3),hidden=False,host='osp-compute-c02-01.tsa402.service-now.com',hostname='test',id=1,image_ref='',info_cache=InstanceInfoCache,instance_type_id=3,kernel_id='',key_data=None,key_name=None,keypairs=<?>,launch_index=0,launched_at=2024-04-01T16:44:29Z,launched_on='osp-compute-c0
2-01.tsa402.service-now.com',locked=False,locked_by=None,memory_mb=1024,metadata={},migration_context=None,new_flavor=None,node='osp-compute-c02-01.tsa402.service-now.com',numa_topology=None,old_flavor=None,os_type=None,pci_devices=<?>,pci_requests=InstancePCIRequests,power_state=1,progress=0,project_id='18321e0d96a74a4ab1ec27394166fb66',ramdisk_id='',reservation_id='r-b59ilff4',resources=None,root_device_name='/dev/vda',root_gb=20,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={boot_roles='member,reader,admin,manager',image_base_image_ref='',image_container_format='bare',image_disk_format='qcow2',image_hw_cdrom_bus='ide',image_hw_disk_bus='virtio',image_hw_input_bus='usb',image_hw_machine_type='pc',image_hw_pointer_model='usbtablet',image_hw_video_model='virtio',image_hw_vif_model='virtio',image_min_disk='20',image_min_ram='0',image_owner_specified.openstack.md5='',image_owner_specified.openstack.object='images/CirrOS-6.2',image_owner_specif
ied.openstack.sha256='',owner_project_name='admin',owner_user_name='admin'},tags=<?>,task_state='migrating',terminated_at=None,trusted_certs=<?>,updated_at=2024-04-01T18:50:15Z,user_data=None,user_id='dea9b2ecc4644988aec805e735f03de3',uuid=b44705f4-6813-42ad-a4a8-a64255e2a6b7,vcpu_model=<?>,vcpus=1,vm_mode=None,vm_state='active')
vif={"id": "702c30b2-f959-489c-894c-0a17e131d0b4", "address":
"fa:16:3e:87:97:92", "network": {"id": "143a8e60-fe8f-462e-ba1e-637ee961c180",
"bridge": "br-int", "label": "admin.local", "subnets": [{"cidr":
"192.168.0.0/24", "dns": [{"address": "10.255.0.108", "type": "dns", "version":
4, "meta": {}}], "gateway": {"address": "192.168.0.1", "type": "gateway",
"version": 4, "meta": {}}, "ips": [{"address": "192.168.0.34", "type": "fixed",
"version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4,
"meta": {"enable_dhcp": true}}], "meta": {"injected": false, "tenant_id":
"18321e0d96a74a4ab1ec27394166fb66", "mtu": 1500, "physical_network": null,
"tunneled": true}}, "type": "ovs", "details": {"port_filter": true,
"connectivity": "l2", "bridge_name": "br-int", "datapath_type": "system"},
"devname": "tap702c30b2-f9", "ovs_interfaceid":
"702c30b2-f959-489c-894c-0a17e131d0b4", "qbh_params": null, "qbg_params": null,
"active": false, "vnic_type": "normal", "profile": {"os_vif_delegation": true},
"preserve_on_delete": false, "delegate_create": true, "meta": {}} virt_type=kvm
get_config
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/vif.py:563
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.682 27905 DEBUG nova.network.os_vif_util [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] Converting VIF {"id":
"702c30b2-f959-489c-894c-0a17e131d0b4", "address": "fa:16:3e:87:97:92",
"network": {"id": "143a8e60-fe8f-462e-ba1e-637ee961c180", "bridge": "br-int",
"label": "admin.local", "subnets": [{"cidr": "192.168.0.0/24", "dns":
[{"address": "10.255.0.108", "type": "dns", "version": 4, "meta": {}}],
"gateway": {"address": "192.168.0.1", "type": "gateway", "version": 4, "meta":
{}}, "ips": [{"address": "192.168.0.34", "type": "fixed", "version": 4, "meta":
{}, "floating_ips": []}], "routes": [], "version": 4, "meta": {"enable_dhcp":
true}}], "meta": {"injected": false, "tenant_id":
"18321e0d96a74a4ab1ec27394166fb66", "mtu": 1500, "physical_network": null,
"tunneled": true}}, "type": "ovs", "details": {"port_filter": true, "
connectivity": "l2", "bridge_name": "br-int", "datapath_type": "system"},
"devname": "tap702c30b2-f9", "ovs_interfaceid":
"702c30b2-f959-489c-894c-0a17e131d0b4", "qbh_params": null, "qbg_params": null,
"active": false, "vnic_type": "normal", "profile": {"os_vif_delegation": true},
"preserve_on_delete": false, "delegate_create": true, "meta": {}}
nova_to_osvif_vif
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/network/os_vif_util.py:511
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.682 27905 DEBUG nova.network.os_vif_util [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] Converted object
VIFOpenVSwitch(active=False,address=fa:16:3e:87:97:92,bridge_name='br-int',has_traffic_filtering=True,id=702c30b2-f959-489c-894c-0a17e131d0b4,network=Network(143a8e60-fe8f-462e-ba1e-637ee961c180),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap702c30b2-f9')
nova_to_osvif_vif
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/network/os_vif_util.py:548
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.683 27905 DEBUG nova.virt.libvirt.migration [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance:
b44705f4-6813-42ad-a4a8-a64255e2a6b7] Updating guest XML with vif config:
<interface type="ethernet">#012 <mac address="fa:16:3e:87:97:92"/>#012 <model
type="virtio"/>#012 <mtu size="1500"/>#012 <target
dev="tap702c30b2-f9"/>#012</interface>#012 _update_vif_xml
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/migration.py:388
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.683 27905 DEBUG nova.virt.libvirt.driver [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance:
b44705f4-6813-42ad-a4a8-a64255e2a6b7] About to invoke the migrate API
_live_migration_operation
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py:10274
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.786 27905 ERROR nova.virt.libvirt.driver [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance:
b44705f4-6813-42ad-a4a8-a64255e2a6b7] Live Migration failure:
(domain_definition):1: Start tag expected, '<' not found#012*
openvswitch.service - Open vSwitch#012^: libvirt.libvirtError:
(domain_definition):1: Start tag expected, '<' not found
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: 2024-04-01
11:50:56.787 27905 DEBUG nova.virt.libvirt.driver [None
req-e8e1cbc3-257a-48e8-9181-b940fa152cac dea9b2ecc4644988aec805e735f03de3
18321e0d96a74a4ab1ec27394166fb66 - - default default] [instance:
b44705f4-6813-42ad-a4a8-a64255e2a6b7] Migration operation thread notification
thread_finished
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py:10632
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: Traceback (most
recent call last):
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/hubs/hub.py",
line 476, in fire_timers
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: timer()
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/hubs/timer.py",
line 59, in __call__
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: cb(*args, **kw)
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/event.py",
line 175, in _do_send
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]:
waiter.switch(result)
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/greenthread.py",
line 221, in main
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: result =
function(*args, **kwargs)
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/utils.py",
line 664, in context_wrapper
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: return func(*args,
**kwargs)
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/driver.py",
line 10287, in _live_migration_operation
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: with
excutils.save_and_reraise_exception():
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]:
self.force_reraise()
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: raise self.value
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/nova/virt/libvirt/guest.py",
line 642, in migrate
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]:
self._domain.migrateToURI3(
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/tpool.py",
line 193, in doit
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: result =
proxy_call(self._autowrap, f, *args, **kwargs)
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/tpool.py",
line 151, in proxy_call
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: rv = execute(f,
*args, **kwargs)
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/tpool.py",
line 132, in execute
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: six.reraise(c, e,
tb)
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/six.py", line
719, in reraise
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: raise value
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/eventlet/tpool.py",
line 86, in tworker
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: rv = meth(*args,
**kwargs)
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: File
"/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/libvirt.py",
line 2168, in migrateToURI3
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: raise
libvirtError('virDomainMigrateToURI3() failed')
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: libvirt.libvirtError:
(domain_definition):1: Start tag expected, '<' not found
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: * openvswitch.service
- Open vSwitch
Apr 1 11:50:56 osp-compute-c02-01 nova-compute[27905]: ^
Apr 1 11:50:57 osp-compute-c01-03 nova-compute[28672]: 2024-04-01
11:50:57.045 28672 DEBUG ovsdbapp.backend.ovs_idl.vlog [-] [POLLIN] on fd 23
__log_wakeup
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/ovs/poller.py:263
Apr 1 11:50:57 osp-compute-c01-01 nova-compute[27897]: 2024-04-01
11:50:57.202 27897 DEBUG oslo_service.periodic_task [None
req-161fba31-483d-4517-ade1-176e066f8c31 - - - - - -] Running periodic task
ComputeManager._instance_usage_audit run_periodic_tasks
/openstack/venvs/nova-28.1.1.dev4/lib64/python3.9/site-packages/oslo_service/periodic_task.py:210
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2059962/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
