Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
On Fri, Mar 01, 2013 at 11:56:00PM +0100, Jiri Denemark wrote: [...] = Getting information about CPU models = Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to be able to query for CPU model details, to find out how it can create a VM that matches what was requested by the user. Current problem: libvirt has a copy of the CPU model definitions on its cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU change. libvirt also assumes that the set of features on each model is always the same on all machine-types, which is not true. Challenge: the resulting CPU features depend on lots of factors, including the machine-type. Workaround: start a paused VM and query for the CPU device information after the CPU was created. I just noticed another problem here, but this gave me an idea that would help solve the enforce error reporting problem: Problem: qemu -machine M -cpu model will create CPU objects where the CPU features are _already_ filtered based on host capabilities. Ah, it seems logical now that you mention it :-) * Using enforce wouldn't solve it, because then QEMU would abort, and QMP would be unavailable. Solution: we could have a CPU object property like removed-features that would have the list of features that were disabled because they are not supported by the host (and would make enforce fail). * This would solve the problem above and also be a machine-friendly way to check for possible enforce errors. * In other words: instead of enforce, libvirt could use check instead of enforce, and before unpausing the VM (or even starting migration), it should first check if the removed-features property is empty. Would that work for you? Yes, that seems like it could work. In fact, it seems much better than using enforce and trying to deal with aborted QEMU. To make the libvirt logic simpler, we could do this: have a force mode (in addition to check/enforce), that wouldn't filter any CPU feature based on host capabilities. This way libvirt wouldn't need any non-trivial logic to combine the f-* flags and removed-features to find out the CPU model details. Then libvirt would need to look only at f-* to find out the CPU model details at probing time (as long as force is used at probing time), and just make sure removed-features is empty before starting the VM (optionally parsing its value or checking the f-* property values, to find out which features are missing exactly). -- Eduardo
Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
On Fri, Mar 01, 2013 at 03:58:18PM -0300, Eduardo Habkost wrote: On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote: Am 01.03.2013 14:12, schrieb Jiri Denemark: On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote: = Listing CPU models = Requirement: libvirt needs to know which CPU models are available to be used with the -cpu option. Current problem: libvirt relies on help output parsing for that. query-cpu-definitions is the QMP command to retrieve values compatible with -cpu. And if libvirt is not using it, I really don't understand why the work of maintaining this crappy interface has been pushed onto us in the first place? There is no reuse between -cpu ? and QMP implementations so it's just extra work, there is no communicated or implemented way to extend the arch_query_cpu_definitions() implementation to become more usable for command line output implementation (e.g., associating a PVR value with the model name for ppc) and, while we're at it, it uses global functions plus a stub rather than a CPUState hook with a no-op default implementation in qom/cpu.c... I have the same questions you have. :-) But my main complaint about query-cpu-definitions is not about the implementation: it's that the interface was introduced without taking into account the requirements of libvirt regarding CPU features. It was found to be not appropriate for what libvirt needs[1], but somehow it got applied anyway. The requirement at the time we did this was just to have a straight adaptation of the stuff libvirt parsed from -help, into QMP format. The query-cpu-definitions command served that purpose fine. It was expected that some of this may be sub-optimal in the long term. This is fine - we can either extend the existing data, or introduce brand new commands to deal with the problems. The key thing was just to get 100% into the QMP world for capabilities at that time, and then iterate to improve things. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
Hi,
After a long time trying to figure out the proper modelling inside QEMU,
I believe the plans are now clearer in QEMU, so it's time to coordinate
more closely with libvirt to try to make use of the new stuff.
I tried to enumerate the libvirt requirements and current problems, and
how we should be able to solve those problems using the X86CPU
subclasses and properties, on the following wiki page:
http://wiki.qemu.org/Features/CPUModels#Interfaces.2Frequirements_from_libvirt
= Ensuring predictable set of guest features =
Requirement: libvirt needs to ensure all features required on the command-line
are present and exposed to the guest.
Current problem: libvirt doesn't use the enforce flag so it can't guarantee
that a given feature will be actually exposed to the guest.
Solution: use the enforce flag on the -cpu option.
Definitely, we plan to start using enforce flag as soon as we have
better CPU probing interface with QEMU. Since libvirt does not currently
consult CPU specs with QEMU, some configurations in fact rely on QEMU
dropping features it can't provide. Of course, that's bad for several
reasons but we don't want such configurations to suddenly stop working.
We want to first fix the CPU specs libvirt creates so that we know they
will work with enforce.
Limitation: no proper machine-friendly interface to report which features
are missing.
Workaround: See querying for host capabilities below.
I doubt we will be ready to start using enforce before the machine
friendly interface is available...
= Listing CPU models =
Requirement: libvirt needs to know which CPU models are available to be used
with the -cpu option.
Current problem: libvirt relies on help output parsing for that.
Solution: use QMP qom-list-types command.
Dependency: X86CPU subclasses.
Limitation: needs a live QEMU process for the query.
No problem, we already run QEMU and use several QMP commands to probe
its capabilities. And qom-list-types is actually one of them. To get
the list of CPU models, we would just call
{
execute: qom-list-types,
arguments: {
implements: X86CPU
}
}
right? What about other non-x86 architectures? Will we need to use
different class name or is there a generic CPU class that could be used
universally?
Solution: use QMP query-cpu-definitions command.
Limitation: needs a live QEMU process for the query.
IIUC, the result of this command will depend on machine type and we
can't use -M none we currently use for probing, right?
== Future plans ==
It would be interesting to get rid of the requirement for a live QEMU process
(with a complete machine being created) to be already running.
Hmm, so is this complete machine needed even for getting CPU models from
qom-list-types or only for querying exact definitions using
query-cpu-definitions command?
Actually, what is query-cpu-definitions supposed to return? Currently it
seems it's just the CPU model names rather than details about all CPU
models. From the command name, one would expect to get more than just
names.
= Getting information about CPU models =
Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to
be able to query for CPU model details, to find out how it can create a VM
that
matches what was requested by the user.
Current problem: libvirt has a copy of the CPU model definitions on its
cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU
change. libvirt also assumes that the set of features on each model is always
the same on all machine-types, which is not true.
Challenge: the resulting CPU features depend on lots of factors, including
the machine-type.
Workaround: start a paused VM and query for the CPU device information
after the CPU was created.
Solution: start a paused VM with no devices, but with the right
machine-type and right CPU model. Use QMP QOM commands to query for CPU
flags (especially the properties starting with the f- prefix).
Dependency: X86CPU feature properties (f-* properties).
Limitation: requires a live QEMU process with the right machine-type/
CPU-model to be started, to make the query.
This would be very useful for ensuring the guest sees the exact same CPU
after it's been migrated or restored from a stored state or a snapshot.
Should we make sure the guest will always see the same CPU even after
shutdown or is it ok if the guest CPU changes a bit on next boot, e.g.,
in case the host kernel was upgraded and is able to provide more
features?
However, probing several CPU definitions for compatibility with
host/kernel/QEMU would be quite inefficient. Although I guess we should
be able to limit doing so only in case a specific API (e.g.,
virConnectCompareCPU or virConnectBaselineCPU) is called,
Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote: = Querying host capabilities = Requirement: libvirt needs to know which feature can really be enabled, before it tries to start a VM, and before it tries to start a live-migration process. The set of available capabilities depend on: • Host CPU (hardware) capabilities; • Kernel capabilities (reported by GET_SUPPORTED_CPUID); • QEMU capabilities; • Specific configuration options (e.g. in-kernel IRQ chip is required for some features). Actually, one more thing. Can any of these requirements change while a host is up and QEMU is not upgraded? I believe, host CPU capabilities can only change when the host starts. Kernel capabilities are a bit less clear since I guess they could possibly change when kvm module is unloaded and loaded back with a different options. QEMU capabilities should only change when different version is installed. And the specific configuration options are the most unclear to me. The reason I'm asking is whether libvirt could run-time cache CPU definitions (including all model details) in the same way we currently cache QEMU capabilities, such as availability of specific QMP commands. Jirka
Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
On Fri, Mar 01, 2013 at 02:12:38PM +0100, Jiri Denemark wrote:
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
Hi,
After a long time trying to figure out the proper modelling inside QEMU,
I believe the plans are now clearer in QEMU, so it's time to coordinate
more closely with libvirt to try to make use of the new stuff.
I tried to enumerate the libvirt requirements and current problems, and
how we should be able to solve those problems using the X86CPU
subclasses and properties, on the following wiki page:
http://wiki.qemu.org/Features/CPUModels#Interfaces.2Frequirements_from_libvirt
= Ensuring predictable set of guest features =
Requirement: libvirt needs to ensure all features required on the
command-line
are present and exposed to the guest.
Current problem: libvirt doesn't use the enforce flag so it can't
guarantee
that a given feature will be actually exposed to the guest.
Solution: use the enforce flag on the -cpu option.
Definitely, we plan to start using enforce flag as soon as we have
better CPU probing interface with QEMU. Since libvirt does not currently
consult CPU specs with QEMU, some configurations in fact rely on QEMU
dropping features it can't provide. Of course, that's bad for several
reasons but we don't want such configurations to suddenly stop working.
We want to first fix the CPU specs libvirt creates so that we know they
will work with enforce.
Also: more important than fixing the CPU definitions from libvirt, is to
ask QEMU for host capabilities and CPU model definitions. The whole
point of this is to solve the CPU model data duplication/synchronization
problems between libvirt and QEMU.
Once you are able to query CPU model definitions on runtime, you don't
even need to make cpu_map.xml agree with QEMU. You can simply ask QEMU
how each model looks like, and remove/add features from the command-line
as necessary, so the resulting VM matches what the user asked for.
Limitation: no proper machine-friendly interface to report which
features
are missing.
Workaround: See querying for host capabilities below.
I doubt we will be ready to start using enforce before the machine
friendly interface is available...
If you query for the -cpu host capabilities first and ensure all
features from a CPU model is available, enforce is supposed to not fail.
I understand that a machine-friendly error reporting for enforce would
be very useful, but note that if enforce fails, it is probably already
too late for libvirt, and that means that what libvirt thinks about host
capabilities and CPU models is already incorrect.
The main problem preventing us from making a machine-friendly interface
is that enforce makes QEMU abort immediately, making us lose the main
machine-friendly communication mechanism, that is QMP.
(But I had an idea to solve that, look for removed-features below for
a description).
= Listing CPU models =
Requirement: libvirt needs to know which CPU models are available to be used
with the -cpu option.
Current problem: libvirt relies on help output parsing for that.
Solution: use QMP qom-list-types command.
Dependency: X86CPU subclasses.
Limitation: needs a live QEMU process for the query.
No problem, we already run QEMU and use several QMP commands to probe
its capabilities. And qom-list-types is actually one of them. To get
the list of CPU models, we would just call
{
execute: qom-list-types,
arguments: {
implements: X86CPU
}
}
right? What about other non-x86 architectures? Will we need to use
different class name or is there a generic CPU class that could be used
universally?
Actually I don't know much about the QMP command syntax and didn't test
it a lot. But that's basically how I think it will look like. Except
that instead of X86CPU, the type name is x86_64-cpu (on
qemu-system-x86_64) and i386-cpu (on qemu-system-i386). Maybe it is
easier to simply use: implements: cpu, abstract: false }.
We may also end up with different CPU model classes for KVM and TCG,
this is still under discussion.
Another caveat: the CPU model class names will be longer than the names
used on the -cpu command-line: something like model-arch-cpu or
model-kvm-arch-cpu.
Solution: use QMP query-cpu-definitions command.
Limitation: needs a live QEMU process for the query.
IIUC, the result of this command will depend on machine type and we
can't use -M none we currently use for probing, right?
The class data don't depend on machine-types. But the resulting CPU
objects may look different depending on machine type.
In other words: you don't need -M to list CPU models, but you need -M if
you want to know which features are going to be available on each model,
exactly.
== Future plans ==
It would be interesting to get rid of the requirement for a live QEMU
Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
On Fri, Mar 01, 2013 at 02:28:37PM +0100, Jiri Denemark wrote: On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote: = Querying host capabilities = Requirement: libvirt needs to know which feature can really be enabled, before it tries to start a VM, and before it tries to start a live-migration process. The set of available capabilities depend on: • Host CPU (hardware) capabilities; • Kernel capabilities (reported by GET_SUPPORTED_CPUID); • QEMU capabilities; • Specific configuration options (e.g. in-kernel IRQ chip is required for some features). Actually, one more thing. Can any of these requirements change while a host is up and QEMU is not upgraded? I believe, host CPU capabilities can only change when the host starts. Kernel capabilities are a bit less clear since I guess they could possibly change when kvm module is unloaded and loaded back with a different options. QEMU capabilities should only change when different version is installed. And the specific configuration options are the most unclear to me. The reason I'm asking is whether libvirt could run-time cache CPU definitions (including all model details) in the same way we currently cache QEMU capabilities, such as availability of specific QMP commands. That's a good question. Let's check each item so I don't forget any detail: • Host CPU (hardware) capabilities; This shouldn't change without a host reboot. • Kernel capabilities (reported by GET_SUPPORTED_CPUID); This may possibly change if the KVM module is unloaded and reloaded with different options, but... I guess we should simply require libvirtd to be restarted if any user does that? • QEMU capabilities; This shouldn't change as long as the QEMU binary doesn't change. • Specific configuration options (e.g. in-kernel IRQ chip is required for some features). This part seems tricky. Currently kernel-irqchip is probably the only option that affects which features are available, but what if other QEMU options affect the set of features too? I believe the answer is to rely on machine-types. I mean: if a new option that affects -cpu host and the set of available CPU features is created, there are two options: 1) Using the default value; 2) Setting the option explicitly. 1) If using the default value, the default will depend on machine-type, so libvirt should already consider machine-type as an option that affects available-CPU-features. 2) If using an explicit value, libvirt should use the explicit value only after being changed to take into account the fact that the option affects available-CPU-features. So, let's add one more item to the list. The set of available capabilities depend on: • Host CPU (hardware) capabilities; • Kernel capabilities (reported by GET_SUPPORTED_CPUID); • QEMU capabilities; • Specific configuration options: • kernel-irqchip (currently affects tsc-deadline and x2apic, but may affect other features in the future) • machine-type (may affect any feature in the future) -- Eduardo
Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
Am 01.03.2013 14:12, schrieb Jiri Denemark:
On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote:
= Listing CPU models =
Requirement: libvirt needs to know which CPU models are available to be used
with the -cpu option.
Current problem: libvirt relies on help output parsing for that.
query-cpu-definitions is the QMP command to retrieve values compatible
with -cpu.
And if libvirt is not using it, I really don't understand why the work
of maintaining this crappy interface has been pushed onto us in the
first place? There is no reuse between -cpu ? and QMP implementations so
it's just extra work, there is no communicated or implemented way to
extend the arch_query_cpu_definitions() implementation to become more
usable for command line output implementation (e.g., associating a PVR
value with the model name for ppc) and, while we're at it, it uses
global functions plus a stub rather than a CPUState hook with a no-op
default implementation in qom/cpu.c...
Solution: use QMP qom-list-types command.
Dependency: X86CPU subclasses.
Limitation: needs a live QEMU process for the query.
No problem, we already run QEMU and use several QMP commands to probe
its capabilities. And qom-list-types is actually one of them. To get
the list of CPU models, we would just call
{
execute: qom-list-types,
arguments: {
implements: X86CPU
}
}
right?
Not quite, this would return abstract types as well, so you'd need to
add abstract: false or so.
And you need to use the type name, not the struct name as argument, i.e.
i386-cpu or x86_64-cpu. Note: This does not always match the
executable name since QOM names are supposed to be verbose (e.g., ppc64
vs. powerpc64-cpu) and some executables are misnamed (sh4 vs. superh-cpu).
For x86 today this will return one type, either i386-cpu or
x86_64-cpu, that's why I have been pushing to implement model
subclasses. There's still some controversial discussions about how this
relates to KVM and TCG changing values of classes.
What about other non-x86 architectures?
For some other architectures like arm this will return the full list of
available classes, but in cortex-a9-arm-cpu format, which is not
guaranteed to be compatible with -cpu but rather with -device where
already applicable.
Will we need to use
different class name or is there a generic CPU class that could be used
universally?
cpu would currently work as well, but the CPU refactorings are
targetting to compile, e.g., arm-cpu and microblaze-cpu types into the
same executable, so cpu may lead to undesired results in the future
depending on what assumptions your code makes.
Regards,
Andreas
--
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote: Am 01.03.2013 14:12, schrieb Jiri Denemark: On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote: = Listing CPU models = Requirement: libvirt needs to know which CPU models are available to be used with the -cpu option. Current problem: libvirt relies on help output parsing for that. query-cpu-definitions is the QMP command to retrieve values compatible with -cpu. Yep, Jiri is wrong here actually. -cpu parsing is what we used to do. With latest libvirt + QEMU we do use query-cpu-definitions and have no help parsing code used at all. Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote: Am 01.03.2013 14:12, schrieb Jiri Denemark: On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote: = Listing CPU models = Requirement: libvirt needs to know which CPU models are available to be used with the -cpu option. Current problem: libvirt relies on help output parsing for that. query-cpu-definitions is the QMP command to retrieve values compatible with -cpu. And if libvirt is not using it, I really don't understand why the work of maintaining this crappy interface has been pushed onto us in the first place? There is no reuse between -cpu ? and QMP implementations so it's just extra work, there is no communicated or implemented way to extend the arch_query_cpu_definitions() implementation to become more usable for command line output implementation (e.g., associating a PVR value with the model name for ppc) and, while we're at it, it uses global functions plus a stub rather than a CPUState hook with a no-op default implementation in qom/cpu.c... I have the same questions you have. :-) But my main complaint about query-cpu-definitions is not about the implementation: it's that the interface was introduced without taking into account the requirements of libvirt regarding CPU features. It was found to be not appropriate for what libvirt needs[1], but somehow it got applied anyway. [1] http://article.gmane.org/gmane.comp.emulators.qemu/164772 -- Eduardo
Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
On Fri, Mar 01, 2013 at 06:34:31PM +, Daniel P. Berrange wrote: On Fri, Mar 01, 2013 at 07:31:46PM +0100, Andreas Färber wrote: Am 01.03.2013 14:12, schrieb Jiri Denemark: On Thu, Feb 21, 2013 at 11:58:18 -0300, Eduardo Habkost wrote: = Listing CPU models = Requirement: libvirt needs to know which CPU models are available to be used with the -cpu option. Current problem: libvirt relies on help output parsing for that. query-cpu-definitions is the QMP command to retrieve values compatible with -cpu. Yep, Jiri is wrong here actually. -cpu parsing is what we used to do. With latest libvirt + QEMU we do use query-cpu-definitions and have no help parsing code used at all. I wrote the above, not Jiri. I will fix the wiki page to note that. -- Eduardo
Re: [Qemu-devel] libvirt-QEMU interfaces for CPU models
On Fri, Mar 01, 2013 at 12:02:07 -0300, Eduardo Habkost wrote: On Fri, Mar 01, 2013 at 02:12:38PM +0100, Jiri Denemark wrote: Definitely, we plan to start using enforce flag as soon as we have better CPU probing interface with QEMU. Since libvirt does not currently consult CPU specs with QEMU, some configurations in fact rely on QEMU dropping features it can't provide. Of course, that's bad for several reasons but we don't want such configurations to suddenly stop working. We want to first fix the CPU specs libvirt creates so that we know they will work with enforce. Also: more important than fixing the CPU definitions from libvirt, is to ask QEMU for host capabilities and CPU model definitions. The whole point of this is to solve the CPU model data duplication/synchronization problems between libvirt and QEMU. Once you are able to query CPU model definitions on runtime, you don't even need to make cpu_map.xml agree with QEMU. You can simply ask QEMU how each model looks like, and remove/add features from the command-line as necessary, so the resulting VM matches what the user asked for. Right, that's what I had in mind. By libvirt providing correct CPU definitions (probably better called the right -cpu command line option) I meant that we need to actually probe QEMU rather than making the definitions on our own from cpu_map.xml and host CPUID. Limitation: no proper machine-friendly interface to report which features are missing. Workaround: See querying for host capabilities below. I doubt we will be ready to start using enforce before the machine friendly interface is available... If you query for the -cpu host capabilities first and ensure all features from a CPU model is available, enforce is supposed to not fail. I understand that a machine-friendly error reporting for enforce would be very useful, but note that if enforce fails, it is probably already too late for libvirt, and that means that what libvirt thinks about host capabilities and CPU models is already incorrect. But we still need to know details about each CPU model so that we can choose the right one. I was trying to say that by the time we have this probing and can start using enforce, the machine friendly reporting could be available as well and the limitation will be gone. == Future plans == It would be interesting to get rid of the requirement for a live QEMU process (with a complete machine being created) to be already running. Hmm, so is this complete machine needed even for getting CPU models from qom-list-types or only for querying exact definitions using query-cpu-definitions command? Maybe complete machine isn't the right expression, here. What I mean is: AFAIK, it is not possible to get a QMP monitor without actually having a machine being created by QEMU (even if it is a machine that will never run). OK, I thought there was something special needed. We already start QEMU in such a way that we can communicate with it using QMP monitor. So this is just a question of using the right machine if we need to know the details about given CPU model. = Getting information about CPU models = Requirement: libvirt uses the predefined CPU models from QEMU, but it needs to be able to query for CPU model details, to find out how it can create a VM that matches what was requested by the user. Current problem: libvirt has a copy of the CPU model definitions on its cpu_map.xml file, and the copy can be out of sync in case CPU models in QEMU change. libvirt also assumes that the set of features on each model is always the same on all machine-types, which is not true. Challenge: the resulting CPU features depend on lots of factors, including the machine-type. Workaround: start a paused VM and query for the CPU device information after the CPU was created. I just noticed another problem here, but this gave me an idea that would help solve the enforce error reporting problem: Problem: qemu -machine M -cpu model will create CPU objects where the CPU features are _already_ filtered based on host capabilities. Ah, it seems logical now that you mention it :-) * Using enforce wouldn't solve it, because then QEMU would abort, and QMP would be unavailable. Solution: we could have a CPU object property like removed-features that would have the list of features that were disabled because they are not supported by the host (and would make enforce fail). * This would solve the problem above and also be a machine-friendly way to check for possible enforce errors. * In other words: instead of enforce, libvirt could use check instead of enforce, and before unpausing the VM (or even starting migration), it should first check if the removed-features property
