Re: [Openstack] Live Migration and LibVirt CPU Mode
- Original Message - > From: "Oisin O'Malley" <oisin.omal...@iocane.com.au> > To: "Robert Van Leeuwen" <rovanleeu...@ebay.com>, > openstack@lists.openstack.org > Sent: Wednesday, July 5, 2017 10:13:48 PM > Subject: Re: [Openstack] Live Migration and LibVirt CPU Mode > > > >> The instance can also be migrated between 2 Westmere hosts. > >> As I outline in another email, I suspect (perhaps incorrectly) as the > >> VM has "cpu mode='host-model'" in its running config, libvirt may be > >> comparing source and destination host CPUs and not guest and destination > >> CPUs. > > > >Indeed it very much seems like it will look at the host cpu-model and not > >the guest since it is complaining about Broadwell CPU features. > >The libvirt documentation also sort of suggests this is the case: > >https://libvirt.org/formatdomain.html#elementsCPU > >“Prior to libvirt 3.2.0 and QEMU 2.9.0 detection of the host CPU model via > >QEMU is not supported. Thus the CPU configuration created using host-model > >may not work as expected.” > > I think this may only apply when initially booting VM with "cpu > mode='host-model'", when it determines compatible CPU features. Though I may > be wrong. I doubled checked this with folks in #openstack-nova and we basically came to the same conclusion and that this is the expected behavior. It's important to choose cpu_mode/cpu_model settings up front based on your environment and requirements. Using host-model works well when all hosts have CPUs of roughly the same generation but when dealing with hosts that span multiple generations it's better to either segregate them (e.g. host aggregates) or choose specify a lowest common denominator baseline using cpu_mode=custom and a specific model. I'm not aware of a safe/guaranteed way to get a running guest from the newer generation host to the older generation host where the guest is using host-model, and even if such a mechanism did exist keep in mind it would also be reliant on the guest not panicing when the CPU exposed to it is swapped out for one with less features. Thanks, Steve ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Live Migration and LibVirt CPU Mode
>> The instance can also be migrated between 2 Westmere hosts.
>> As I outline in another email, I suspect (perhaps incorrectly) as the
>> VM has "cpu mode='host-model'" in its running config, libvirt may be
>> comparing source and destination host CPUs and not guest and destination
>> CPUs.
>
>Indeed it very much seems like it will look at the host cpu-model and not the
>guest since it is complaining about Broadwell CPU features.
>The libvirt documentation also sort of suggests this is the case:
>https://libvirt.org/formatdomain.html#elementsCPU
>“Prior to libvirt 3.2.0 and QEMU 2.9.0 detection of the host CPU model via
>QEMU is not supported. Thus the CPU configuration created using host-model may
>not work as expected.”
I think this may only apply when initially booting VM with "cpu
mode='host-model'", when it determines compatible CPU features. Though I may be
wrong.
>You can have a look cheating your way around it.
>There is some code which removes the checks which you might be able to (ab)use:
>https://bugs.launchpad.net/nova/+bug/1588003
I don't think this will apply either as the error in the traceback is coming
from LibVirt itself not Nova, which would imply Nova's pre-migration
compatibility checks have passed and it attempted and failed to call LibVirt to
migrate the VM.
nova-compute: Traceback (most recent call last):
nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py",
line 457, in fire_timers
nova-compute: timer()
nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py",
line 58, in __call__
nova-compute: cb(*args, **kw)
nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/event.py", line
168, in _do_send
nova-compute: waiter.switch(result)
nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py",
line 214, in main
nova-compute: result = function(*args, **kwargs)
nova-compute: File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1066,
in context_wrapper
nova-compute: return func(*args, **kwargs)
nova-compute: File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5911, in
_live_migration_operation
nova-compute: instance=instance)
nova-compute: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
nova-compute: self.force_reraise()
nova-compute: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
nova-compute: six.reraise(self.type_, self.value, self.tb)
nova-compute: File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5907, in
_live_migration_operation
nova-compute: bandwidth=CONF.libvirt.live_migration_bandwidth)
nova-compute: File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 568, in
migrate
nova-compute: destination, params=params, flags=flags)
nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line
186, in doit
nova-compute: result = proxy_call(self._autowrap, f, *args, **kwargs)
nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line
144, in proxy_call
nova-compute: rv = execute(f, *args, **kwargs)
nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line
125, in execute
nova-compute: six.reraise(c, e, tb)
nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line
83, in tworker
nova-compute: rv = meth(*args, **kwargs)
nova-compute: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1939,
in migrateToURI3
nova-compute: if ret == -1: raise libvirtError ('virDomainMigrateToURI3()
failed', dom=self)
nova-compute: libvirtError: unsupported configuration: guest and host CPU are
not compatible: Host CPU does not provide required features: fma, x2apic,
movbe, tsc-deadline, xs
>You might want to double-check if a (hard)-rebooted migrated (with the
>host-model config) instances on the Broadwell compute node does not get a
>Broadwell cpu before modifying this ;)
It doesn't, as the Broadwell hosts nova .conf has the following;
cpu_mode = custom
cpu_model = Westmere
Oisin O'Malley
Systems Engineer
Iocane Pty Ltd
763 South Road
Black Forest SA 5035
Office:+61 (8) 8413 1010
Fax:+61 (8) 8231 2050
Email:oisin.omal...@iocane.com.au
Web:www.iocane.com.au
Better for business
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Live Migration and LibVirt CPU Mode
> The instance can also be migrated between 2 Westmere hosts. > As I outline in another email, I suspect (perhaps incorrectly) as the VM has > "cpu mode='host-model'" in its running config, > libvirt may be comparing source and destination host CPUs and not guest and > destination CPUs. Indeed it very much seems like it will look at the host cpu-model and not the guest since it is complaining about Broadwell CPU features. The libvirt documentation also sort of suggests this is the case: https://libvirt.org/formatdomain.html#elementsCPU “Prior to libvirt 3.2.0 and QEMU 2.9.0 detection of the host CPU model via QEMU is not supported. Thus the CPU configuration created using host-model may not work as expected.” You can have a look cheating your way around it. There is some code which removes the checks which you might be able to (ab)use: https://bugs.launchpad.net/nova/+bug/1588003 You might want to double-check if a (hard)-rebooted migrated (with the host-model config) instances on the Broadwell compute node does not get a Broadwell cpu before modifying this ;) Cheers, Robert van Leeuwen ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Live Migration and LibVirt CPU Mode
>Hello.. >can you send cpu info(/proc/cpuinfo or dmidecode)? Westmere host /proc/cpuinfo: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU E5640 @ 2.67GHz stepping: 2 microcode : 0x1d cpu MHz : 2666.000 cache size : 12288 KB physical id : 1 siblings: 8 core id : 0 cpu cores : 4 apicid : 32 initial apicid : 32 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dtherm tpr_shadow vnmi flexpriority ept vpid bogomips: 5333.43 clflush size: 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: Westmere host libvirt CPU capabilities; x86_64 Westmere Intel x86_64 Broadwell Intel Oisin O'Malley Systems Engineer Iocane Pty Ltd 763 South Road Black Forest SA 5035 Office:+61 (8) 8413 1010 Fax:+61 (8) 8231 2050 Email:oisin.omal...@iocane.com.au Web:www.iocane.com.au Better for business ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Live Migration and LibVirt CPU Mode
Hello.. can you send cpu info(/proc/cpuinfo or dmidecode)? see => https://wiki.libvirt.org/page/Libvirt_identifies_host_processor_as_a_different_model_from_the_hardware_documentation https://serverfault.com/questions/585791/which-cpu-model-to-set-for-guest-kvm-when-processor-is-ivy-bridge libvirt linux => /usr/share/libvirt/cpu_map.xml qemu => qemu-system-x86_64 --cpu ? x86 qemu64 QEMU Virtual CPU version 2.5+ x86 phenom AMD Phenom(tm) 9550 Quad-Core Processor x86 core2duo Intel(R) Core(TM)2 Duo CPU T7700 @ 2.40GHz x86kvm64 Common KVM processor x86 qemu32 QEMU Virtual CPU version 2.5+ x86kvm32 Common 32-bit KVM processor x86 coreduo Genuine Intel(R) CPU T2600 @ 2.16GHz x86 486 x86 pentium x86 pentium2 x86 pentium3 x86 athlon QEMU Virtual CPU version 2.5+ x86 n270 Intel(R) Atom(TM) CPU N270 @ 1.60GHz x86 Conroe Intel Celeron_4x0 (Conroe/Merom Class Core 2) x86 Penryn Intel Core 2 Duo P9xxx (Penryn Class Core 2) x86 Nehalem Intel Core i7 9xx (Nehalem Class Core i7) x86 *Westmere Westmere E56xx/L56xx/X56xx (Nehalem-C)* x86 SandyBridge Intel Xeon E312xx (Sandy Bridge) x86IvyBridge Intel Xeon E3-12xx v2 (Ivy Bridge) x86Haswell-noTSX Intel Core Processor (Haswell, no TSX) x86 Haswell Intel Core Processor (Haswell) x86 Broadwell-noTSX Intel Core Processor (Broadwell, no TSX) x86Broadwell Intel Core Processor (Broadwell) x86 Opteron_G1 AMD Opteron 240 (Gen 1 Class Opteron) x86 Opteron_G2 AMD Opteron 22xx (Gen 2 Class Opteron) x86 Opteron_G3 AMD Opteron 23xx (Gen 3 Class Opteron) x86 Opteron_G4 AMD Opteron 62xx class CPU x86 Opteron_G5 AMD Opteron 63xx class CPU x86 host KVM processor with all supported host features (only available in KVM mode) Regards. On 05/07/17 01:05, Oisin O'Malley wrote: Hi, Thanks for the suggestion, but it’s still not working. I have changed both nodes to use the Westmere CPU model, which is the lowest common CPU instruction set. The following is in the nova.conf on both nodes; cpu_mode = custom cpu_model = Westmere libvirt_cpu_mode = custom libvirt_cpu_model = Westmere I'm beginning to suspect that because the VM has "cpu mode='host-model'" in its xml CPU definition, libvirt is incorrectly comparing the current host CPU to the target host instead of comparing the guest CPU to the target host CPU. This would be consistent with the error message which states that target host (Westmere) isn't compatible with the 'Broadwell-noTSX' CPU model, even though the VM is configured as Westmere (Westmere). Original Error Message; Host CPU does not provide required features: fma, x2apic, movbe, tsc-deadline, xsave, osxsave, avx, f16c, rdrand, fsgsbase, tsc_adjust, bmi1, hle, avx2, smep, bmi2, erms, invpcid, rtm, rdseed, adx, smap, xsaveopt, abm, 3dnowprefetch; try using 'Broadwell-noTSX' CPU model As mentioned previously, when the VM is booted on a compute node with "cpu_model = Westmere" in its nova.conf, the VMs XML CPU definition contains the following and DOES live migrate; Westmere Therefore it appears if a VM XML CPU definition contains "cpu mode='host-model'" the cpu model is ignored and the 2 host CPU models are compared. And if "cpu mode='custom'", guest and target CPU models are compared. This would also be consistent with the fact I can migrate from a Westmere to a Broadwell, as Broadwell is a superset of Westmere but not the other way around. Any suggestions with dealing with this? Rebooting all hosted instances isn't really an option for us. Regards, Oisin From: rag.open@openmailbox.org [mailto:rag.open@openmailbox.org] Sent: Tuesday, 4 July 2017 9:41 AM To: Oisin O'Malley <oisin.omal...@iocane.com.au> Subject: Re: [Openstack] Live Migration and LibVirt CPU Mode Hello Malley, excuse me for delay... for resolve your problem consider change architecture in 2 compute nodes.. compute1 cat /proc/cpuinfo compute2 cat /proc/cpuinfo search middle architecture for this processor.. my example in my situation 1º compute model name: AMD Phenom(tm) II X6 1090T Processor 2º compute model name: AMD A8-7600 Radeon R7, 10 Compute Cores 4C+6G and de nova in 2 compute: libvirt_cpu_mode = custom cpu_mode = custom libvirt_cpu_model=Opteron_G3 cpu_model=Opteron_G3 Custom: If your nova.conf file contains cpu_mode=custom, you can explicitly specify one of the supported named models using the cpu_model configuration option. For example, to configure the KVM guests to expose >Nehalem CPUs, your nova.conf file should contain Nova cpu that's all. Search compatible architecture Oisin O'Malley Systems Engineer Iocane Pty Ltd 763 South Road Black Forest SA 5035 Office:+61 (8) 8413 1010 Fax:+61 (8) 8231 2050 Email:oisin.omal..
Re: [Openstack] Live Migration and LibVirt CPU Mode
Hi, Thanks for the suggestion, but it’s still not working. I have changed both nodes to use the Westmere CPU model, which is the lowest common CPU instruction set. The following is in the nova.conf on both nodes; cpu_mode = custom cpu_model = Westmere libvirt_cpu_mode = custom libvirt_cpu_model = Westmere I'm beginning to suspect that because the VM has "cpu mode='host-model'" in its xml CPU definition, libvirt is incorrectly comparing the current host CPU to the target host instead of comparing the guest CPU to the target host CPU. This would be consistent with the error message which states that target host (Westmere) isn't compatible with the 'Broadwell-noTSX' CPU model, even though the VM is configured as Westmere (Westmere). Original Error Message; Host CPU does not provide required features: fma, x2apic, movbe, tsc-deadline, xsave, osxsave, avx, f16c, rdrand, fsgsbase, tsc_adjust, bmi1, hle, avx2, smep, bmi2, erms, invpcid, rtm, rdseed, adx, smap, xsaveopt, abm, 3dnowprefetch; try using 'Broadwell-noTSX' CPU model As mentioned previously, when the VM is booted on a compute node with "cpu_model = Westmere" in its nova.conf, the VMs XML CPU definition contains the following and DOES live migrate; Westmere Therefore it appears if a VM XML CPU definition contains "cpu mode='host-model'" the cpu model is ignored and the 2 host CPU models are compared. And if "cpu mode='custom'", guest and target CPU models are compared. This would also be consistent with the fact I can migrate from a Westmere to a Broadwell, as Broadwell is a superset of Westmere but not the other way around. Any suggestions with dealing with this? Rebooting all hosted instances isn't really an option for us. Regards, Oisin >From: rag.open@openmailbox.org [mailto:rag.open@openmailbox.org] >Sent: Tuesday, 4 July 2017 9:41 AM >To: Oisin O'Malley <oisin.omal...@iocane.com.au> >Subject: Re: [Openstack] Live Migration and LibVirt CPU Mode > >Hello Malley, excuse me for delay... > >for resolve your problem consider change architecture in 2 compute nodes.. > >compute1 > >cat /proc/cpuinfo >compute2 > >cat /proc/cpuinfo > >search middle architecture for this processor.. >my example in my situation > >1º compute >model name: AMD Phenom(tm) II X6 1090T Processor >2º compute >model name: AMD A8-7600 Radeon R7, 10 Compute Cores 4C+6G > >and de nova in 2 compute: >libvirt_cpu_mode = custom >cpu_mode = custom >libvirt_cpu_model=Opteron_G3 >cpu_model=Opteron_G3 > >Custom: > >If your nova.conf file contains cpu_mode=custom, you can explicitly specify >one of the supported named models using the cpu_model configuration option. >For example, to configure the KVM guests to expose >Nehalem CPUs, your >nova.conf file should contain > >Nova cpu > >that's all. Search compatible architecture Oisin O'Malley Systems Engineer Iocane Pty Ltd 763 South Road Black Forest SA 5035 Office:+61 (8) 8413 1010 Fax:+61 (8) 8231 2050 Email:oisin.omal...@iocane.com.au Web:www.iocane.com.au Better for business ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Live Migration and LibVirt CPU Mode
> I'm encountering an issue with live migrating between 2 Compute nodes with > different CPU models. Host A has a Westmere CPU and host B has a Broadwell. That’s expected unless you specify the least common denominator with the cpu model, in this case Westmere, when you boot the instances. > I can migrate an instance from the Westmere host A to the Broadwell host B Yes, you can move an instance from a host with less cpu features to more cpu features as long as you do not configure cpu-passthrough AFAIK. > but when I attempt to live migrate a VM from the Broadwell host B to the > older Westmere host I get the following error; > Live Migration failure: unsupported configuration: guest and host CPU are not > compatible: > Host CPU does not provide required features: fma, x2apic, movbe, > tsc-deadline, xsave, osxsave, avx, …. That’s because the instances booted with those extra features enabled. (you can also see feature when grepping for the qemu process in the processlist, you will see the configured cpu in the qemu commandline) The Westmere CPU does not support avx/avx2 etc. >How can I avoid this error, without rebooting any VMs? You cannot do that for any instance which is already running with the extra cpu flags. Basically, you would need to remove cpu-features on the fly when live-migrating. That’s not possible. > Host B with the Broadwell CPU has the libvirt CPU masking enabled in its > nova.conf; > [libvirt] > cpu_mode = custom >cpu_model = Westmere > >Host A with the Westmere CPU, never had the cpu_mode set in nova.conf but > as its KVM it should default to the following; >[libvirt] >cpu_mode = host-model > According the docs :"host-model" - this causes libvirt to identify the named CPU model which most closely matches the host from the above list, and then request additional CPU flags to complete the match. If you want full live migration compatibility across different cpu generations I would recommend to set the cpu mode and model everywhere to the same values. > Interestingly once a host is booted on Host B and has the above CPU config, > it can be migrated back to Host A without any error. I would expect that, as long as the destination CPU has all the cpu features where the qemu process was started with (in this case Westmere profile) it should work. There have been a couple of bugs in the past with the comparison of cpu features but I think what you are seeing is expected behavior. Note that there is quite a bit of change between Westmere and Broadwell and you might really want those extra cpu features. e.g. avx2 can seriously improve SSL offloading (200% improvement with the correct cypher suites) Cheers, Robert van Leeuwen ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] Live Migration and LibVirt CPU Mode
Hi All, I'm encountering an issue with live migrating between 2 Compute nodes with different CPU models. Host A has a Westmere CPU and host B has a Broadwell. I can migrate an instance from the Westmere host A to the Broadwell host B, but when I attempt to live migrate a VM from the Broadwell host B to the older Westmere host I get the following error; Live Migration failure: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: fma, x2apic, movbe, tsc-deadline, xsave, osxsave, avx, f16c, rdrand, fsgsbase, tsc_adjust, bmi1, hle, avx2, smep, bmi2, erms, invpcid, rtm, rdseed, adx, smap, xsaveopt, abm, 3dnowprefetch; try using 'Broadwell-noTSX' CPU model How can I avoid this error, without rebooting any VMs? Host B with the Broadwell CPU has the libvirt CPU masking enabled in its nova.conf; ... [libvirt] cpu_mode = custom cpu_model = Westmere Host A with the Westmere CPU, never had the cpu_mode set in nova.conf but as its KVM it should default to the following; [libvirt] cpu_mode = host-model The "host-model" cpu_mode appears to enable more CPU features on the guest the then what is enabled by cpu_model = Westmere. For instance, when a VM is started on Host A (Westmere CPU & cpu_mode = host-model) the VM get the following CPU configuration; Westmere Intel But when the instance is booted on Host B (Broadwell CPU & cpu_model = Westmere) Westmere Interestingly once a host is booted on Host B and has the above CPU config, it can be migrated back to Host A without any error. Host A Capabilities; x86_64 Westmere Intel Host B Capabilities; x86_64 Broadwell Intel Oisin O'Malley Systems Engineer Iocane Pty Ltd 763 South Road Black Forest SA 5035 Office:+61 (8) 8413 1010 Fax:+61 (8) 8231 2050 Email:oisin.omal...@iocane.com.au Web:www.iocane.com.au Better for business ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
