Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 11:23:16PM +0200, Michael S. Tsirkin wrote: So any single CPU flag now needs to be added in - kvm - qemu - libvirt This is in fact already the case, and it will also possibly need to be added to openstack too. Next thing libvirt will decide it's a policy thing and so needs to be pushed up to openstack. In fact openstack would really like it if we did exactly that, but even just having CPUs versioned separately from machine types would be a big step forward as far as openstack is concerned. The openstack schedular does not have full visibility into the way the guest is going to be configured by libvirt/QEMU, in particular it does not know anything about machine type that will be used by the guest. The compute hosts report what CPU features they can support, and the user / admin will be able to express what CPU model and/or features they desire their guest to run with, and the schedular has to match that up to decide on hosts to use. If the CPU QEMU machine type used can alter what the CPU model means in terms of features, then the schedling decisions OpenStack is making are going to be wrong some portion of the time. So from the POV of the OpenStack schedular, we'd much rather have CPU models versioned explicitly so their semantics do not change behind our back based on machine types. OpenStack is also looking at how to present a consistent description of CPU models to users, which is independant of the cloud. Since libvirt/QEMU doesn't allow 100% control of the CPU model, OpenStack is likely going to have to define some kind of manual mapping of its own. We should just figure out what you want to do and support it in QEMU. Main thing is versioned CPU models with fixed semantics that don't magically change based on other aspects of VM configuration, such as the machine type. This could be accomplished by QEMU alone. Following on from that though, there's two other aspects which we'd like to address. First, be able to present a consistent set of CPU models across all hypervisors libvirt manages, regardless of type or version. This is a key reason why we have always maintained our own CPU model database, even though it duplicates QEMU's. More interesting is the question of host passthrough. We have two modes for that - either 'host-model' or 'host-passthrough'. The 'host-passthrough' option is something that explicitly maps to QEMU's '-cpu host'. This is good because it exposes 100% of the host CPU to the guest. This is bad because it then prevents use of migration in general, unless both machines are 100% identical - libvirt just blocks it rather than trying todo such a fine grained host CPU check. For that reason we have 'host-model', which is supposed to be essentially the same thing instead of '-cpu host' we explicitly list all the features alongside a named model. Since we control exactly what the guest is being given, we can permit guests with 'host-model' to be migrated, even if the destination host is a superset of the source host, we know the guest won't suddenly see a model change after migration. Currently we are limited though, as we can only express the CPU features - we cannot expose the other aspects like level, xlevel, etc. So our 'host-model' is not quite as perfect a match as '-cpu host' is. The '-cpu custom' would help us getting a better match for 'host-model' by allowing these extra things to be configured. Are there many examples where a single flag used to work and then stopped? I would say such a change is problematic anyway: not everyone uses libvirt, you are breaking things for people that run -M pc. IMHO -M pc is not supposed to mean can break at any time. Well 'pc' is an unversioned machine type, so it explicitly is said to break at any time - users/apps are supposed to translate that into a versioned type if they want a guarantee of no breakage. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 14:32:00 +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. QEMU provides stable ABI for x86 CPUs only if you use -cpu ...,enforce. Without enforce the CPU may change everytime a domain is started or migrated. A small example: let's say a CPU model called Model includes feature xyz; when QEMU is started with -cpu Model (no enforce) on a host which supports xyz, the guest OS will see a CPU with xyz, but when you migrate it to a host which does not support xyz, QEMU will just silently drop xyz. In other words, we need to use enforce to make sure CPU ABI does not change. But the problem is we can't use enforce because we don't know how a specific CPU model looks like for a given machine type. Remember, while libvirt allows users to explicitly ask for a specific CPU model and features, it also has a mode when libvirt itself computes the right CPU model and features. And this is impossible with enforce without us knowing all details about CPU models. So there are two possible ways to address this: 1. enhance QEMU to give us all we need - either by providing commands that would do all the computations (CPU model comparison, intersections or denominator, something like -cpu best) - or provide a way to probe for all (currently 700+) combinations of a CPU model and a machine type without actually having to start QEMU with each CPU and a machine type separately 2. manage CPU models in libvirt (aka -cpu custom) During the past several years Eduardo tried to do (1) without getting anywhere close to something that QEMU would be willing to accept. On the other hand (2) is a pretty minimal change to QEMU and is more flexible than (1) because it allows CPU model versions to be decoupled from machine types (but this was already discussed a lot in the other emails in this thread). Jirka
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Wed, Jun 24, 2015 at 11:20:50AM +0200, Jiri Denemark wrote: On Tue, Jun 23, 2015 at 14:32:00 +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. QEMU provides stable ABI for x86 CPUs only if you use -cpu ...,enforce. Without enforce the CPU may change everytime a domain is started or migrated. A small example: let's say a CPU model called Model includes feature xyz; when QEMU is started with -cpu Model (no enforce) on a host which supports xyz, the guest OS will see a CPU with xyz, but when you migrate it to a host which does not support xyz, QEMU will just silently drop xyz. In other words, we need to use enforce to make sure CPU ABI does not change. Are there really many examples like this? Could someone supply some examples? Eduardo gave examples of CPU changes across machine types but I haven't seen examples where we would break runnability. But the problem is we can't use enforce because we don't know how a specific CPU model looks like for a given machine type. Remember, while libvirt allows users to explicitly ask for a specific CPU model and features, it also has a mode when libvirt itself computes the right CPU model and features. And this is impossible with enforce without us knowing all details about CPU models. So there are two possible ways to address this: 1. enhance QEMU to give us all we need - either by providing commands that would do all the computations (CPU model comparison, intersections or denominator, something like -cpu best) - or provide a way to probe for all (currently 700+) combinations of a CPU model and a machine type without actually having to start QEMU with each CPU and a machine type separately 2. manage CPU models in libvirt (aka -cpu custom) During the past several years Eduardo tried to do (1) without getting anywhere close to something that QEMU would be willing to accept. And the reason, presumably, is because it's a hard problem to solve. Why is it easier to solve at the libvirt level? On the other hand (2) is a pretty minimal change to QEMU and is more flexible than (1) because it allows CPU model versions to be decoupled from machine types (but this was already discussed a lot in the other emails in this thread). Jirka I'm fine with the change itself, it's useful e.g. for testing. But how is it a solution for libvirt's problems? What is libvirt going to do in the above cases? -- MST
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Wed, Jun 24, 2015 at 09:52:09AM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 11:23:16PM +0200, Michael S. Tsirkin wrote: So any single CPU flag now needs to be added in - kvm - qemu - libvirt This is in fact already the case, and it will also possibly need to be added to openstack too. Next thing libvirt will decide it's a policy thing and so needs to be pushed up to openstack. In fact openstack would really like it if we did exactly that, but even just having CPUs versioned separately from machine types would be a big step forward as far as openstack is concerned. The openstack schedular does not have full visibility into the way the guest is going to be configured by libvirt/QEMU, in particular it does not know anything about machine type that will be used by the guest. The compute hosts report what CPU features they can support, and the user / admin will be able to express what CPU model and/or features they desire their guest to run with, and the schedular has to match that up to decide on hosts to use. If the CPU QEMU machine type used can alter what the CPU model means in terms of features, then the schedling decisions OpenStack is making are going to be wrong some portion of the time. Is this all just theoretical, or are there real examples where things stop running? I keep hearing feature xyz and it's impossible to argue reasonably about that IMHO. So from the POV of the OpenStack schedular, we'd much rather have CPU models versioned explicitly so their semantics do not change behind our back based on machine types. OpenStack is also looking at how to present a consistent description of CPU models to users, which is independant of the cloud. Since libvirt/QEMU doesn't allow 100% control of the CPU model, OpenStack is likely going to have to define some kind of manual mapping of its own. We should just figure out what you want to do and support it in QEMU. Main thing is versioned CPU models with fixed semantics that don't magically change based on other aspects of VM configuration, such as the machine type. This could be accomplished by QEMU alone. Following on from that though, there's two other aspects which we'd like to address. First, be able to present a consistent set of CPU models across all hypervisors libvirt manages, regardless of type or version. This is a key reason why we have always maintained our own CPU model database, even though it duplicates QEMU's. Do you also want to migrate that? If yes, the problem definitely becomes more than just CPU specific. More interesting is the question of host passthrough. We have two modes for that - either 'host-model' or 'host-passthrough'. The 'host-passthrough' option is something that explicitly maps to QEMU's '-cpu host'. This is good because it exposes 100% of the host CPU to the guest. This is bad because it then prevents use of migration in general, unless both machines are 100% identical - libvirt just blocks it rather than trying todo such a fine grained host CPU check. For that reason we have 'host-model', which is supposed to be essentially the same thing instead of '-cpu host' we explicitly list all the features alongside a named model. Since we control exactly what the guest is being given, we can permit guests with 'host-model' to be migrated, even if the destination host is a superset of the source host, we know the guest won't suddenly see a model change after migration. Currently we are limited though, as we can only express the CPU features - we cannot expose the other aspects like level, xlevel, etc. So our 'host-model' is not quite as perfect a match as '-cpu host' is. The '-cpu custom' would help us getting a better match for 'host-model' by allowing these extra things to be configured. This duplicates code from QEMU though. It looks like we need a tool to get the legal CPUs that can run on the given host? Would be easy to add, reusing QEMU codebase. Maybe make it a new QEMU flag. Are there many examples where a single flag used to work and then stopped? I would say such a change is problematic anyway: not everyone uses libvirt, you are breaking things for people that run -M pc. IMHO -M pc is not supposed to mean can break at any time. Well 'pc' is an unversioned machine type, so it explicitly is said to break at any time - users/apps are supposed to translate that into a versioned type if they want a guarantee of no breakage. Regards, Daniel That's because you are looking at it from libvirt perspective. From QEMU command line, since pc is the default, this makes no sense IMHO: look up usage advice on the internet, and you will see no one specifies a machine type. -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o-
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Wed, Jun 24, 2015 at 12:21:57PM +0200, Michael S. Tsirkin wrote: On Wed, Jun 24, 2015 at 11:20:50AM +0200, Jiri Denemark wrote: On Tue, Jun 23, 2015 at 14:32:00 +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. QEMU provides stable ABI for x86 CPUs only if you use -cpu ...,enforce. Without enforce the CPU may change everytime a domain is started or migrated. A small example: let's say a CPU model called Model includes feature xyz; when QEMU is started with -cpu Model (no enforce) on a host which supports xyz, the guest OS will see a CPU with xyz, but when you migrate it to a host which does not support xyz, QEMU will just silently drop xyz. In other words, we need to use enforce to make sure CPU ABI does not change. Are there really many examples like this? Could someone supply some examples? Eduardo gave examples of CPU changes across machine types but I haven't seen examples where we would break runnability. But the problem is we can't use enforce because we don't know how a specific CPU model looks like for a given machine type. Remember, while libvirt allows users to explicitly ask for a specific CPU model and features, it also has a mode when libvirt itself computes the right CPU model and features. And this is impossible with enforce without us knowing all details about CPU models. So there are two possible ways to address this: 1. enhance QEMU to give us all we need - either by providing commands that would do all the computations (CPU model comparison, intersections or denominator, something like -cpu best) - or provide a way to probe for all (currently 700+) combinations of a CPU model and a machine type without actually having to start QEMU with each CPU and a machine type separately 2. manage CPU models in libvirt (aka -cpu custom) During the past several years Eduardo tried to do (1) without getting anywhere close to something that QEMU would be willing to accept. And the reason, presumably, is because it's a hard problem to solve. Why is it easier to solve at the libvirt level? One of the main reasons it is hard is because QEMU machine types are not statically introspectable - you have to actually instantiate the machine type to determine what config it produces. This is ultimately a limitation of QOM, and while it could be fixed it would be a pretty significant design change for QEMU at this point. So the reason it would be simpler in libvirt is that we would not have any need to attempt such introspection - the data we need would immediately available to libvirt in the format it needs to use it in. The OpenStack scheduling example I mentioned elsewhere is another reason where the current scheme causes pain - the point at which OpenStack wants to make decisions about host/guest CPU compatibility, we don't even have a guest configuration available yet, so we don't know what machine type we'd want to use, and QEMU isn't even installed on the hosts doing this decision making. Currently OpenStack just has to pretend that CPU models don't change based on machine type. Most of the time we'll be lucky and that won't hurt us, but obviously it is not a desirable thing to have todo. On the other hand (2) is a pretty minimal change to QEMU and is more flexible than (1) because it allows CPU model versions to be decoupled from machine types (but this was already discussed a lot in the other emails in this thread). Jirka I'm fine with the change itself, it's useful e.g. for testing. But how is it a solution for libvirt's problems? What is libvirt going to do in the above cases? Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On 24/06/2015 12:21, Michael S. Tsirkin wrote: QEMU provides stable ABI for x86 CPUs only if you use -cpu ...,enforce. Without enforce the CPU may change everytime a domain is started or migrated. A small example: let's say a CPU model called Model includes feature xyz; when QEMU is started with -cpu Model (no enforce) on a host which supports xyz, the guest OS will see a CPU with xyz, but when you migrate it to a host which does not support xyz, QEMU will just silently drop xyz. In other words, we need to use enforce to make sure CPU ABI does not change. Are there really many examples like this? Could someone supply some examples? Eduardo gave examples of CPU changes across machine types but I haven't seen examples where we would break runnability. Same here, and I would be quite surprised of seeing any. Except perhaps when we introduced x2apic: that would have broken kernels 2.6.32, but I hope we can ignore those. At least as far as I've maintained KVM, we've not added new models to QEMU after KVM's kernel support was added. The only case I can imagine is that the kernel is ancient so it doesn't have anything newer than say IvyBridge, while QEMU is new. If the user starts a Haswell VM on a SandyBridge host without enforce, you will have a problem when you reboot and get a newer kernel, but this is independent of machine types. We have added new features in exactly three cases: 1) F16 and RDRAND to {Haswell,Broadwell} in 2.3; 2) MOVBE to n270 in 1.5; 3) PCLMULQDQ to Westmere also in 1.5. In all three cases, libvirt is still buggy and doesn't let you use the features if you have the appropriate host. We were close to breaking libvirt's expectations when we wanted to remove TSX from Haswell/Broadwell, but in the end we did it right by adding separate CPU models and no final release broke libvirt. So, what is the broken case? And BTW, what is _exactly_ preventing libvirt from using enforce for models other than qemu32/qemu64? Paolo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Wed, Jun 24, 2015 at 11:31:37AM +0100, Daniel P. Berrange wrote: On Wed, Jun 24, 2015 at 12:21:57PM +0200, Michael S. Tsirkin wrote: On Wed, Jun 24, 2015 at 11:20:50AM +0200, Jiri Denemark wrote: On Tue, Jun 23, 2015 at 14:32:00 +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. QEMU provides stable ABI for x86 CPUs only if you use -cpu ...,enforce. Without enforce the CPU may change everytime a domain is started or migrated. A small example: let's say a CPU model called Model includes feature xyz; when QEMU is started with -cpu Model (no enforce) on a host which supports xyz, the guest OS will see a CPU with xyz, but when you migrate it to a host which does not support xyz, QEMU will just silently drop xyz. In other words, we need to use enforce to make sure CPU ABI does not change. Are there really many examples like this? Could someone supply some examples? Eduardo gave examples of CPU changes across machine types but I haven't seen examples where we would break runnability. ^^^ ??? But the problem is we can't use enforce because we don't know how a specific CPU model looks like for a given machine type. Remember, while libvirt allows users to explicitly ask for a specific CPU model and features, it also has a mode when libvirt itself computes the right CPU model and features. And this is impossible with enforce without us knowing all details about CPU models. So there are two possible ways to address this: 1. enhance QEMU to give us all we need - either by providing commands that would do all the computations (CPU model comparison, intersections or denominator, something like -cpu best) - or provide a way to probe for all (currently 700+) combinations of a CPU model and a machine type without actually having to start QEMU with each CPU and a machine type separately 2. manage CPU models in libvirt (aka -cpu custom) During the past several years Eduardo tried to do (1) without getting anywhere close to something that QEMU would be willing to accept. And the reason, presumably, is because it's a hard problem to solve. Why is it easier to solve at the libvirt level? One of the main reasons it is hard is because QEMU machine types are not statically introspectable - you have to actually instantiate the machine type to determine what config it produces. This is ultimately a limitation of QOM, and while it could be fixed it would be a pretty significant design change for QEMU at this point. So the reason it would be simpler in libvirt is that we would not have any need to attempt such introspection - the data we need would immediately available to libvirt in the format it needs to use it in. Why do you want to poke at QEMU machine types? Looks like a small utility printing list of CPU types compatible with the host should be enough. The OpenStack scheduling example I mentioned elsewhere is another reason where the current scheme causes pain - the point at which OpenStack wants to make decisions about host/guest CPU compatibility, we don't even have a guest configuration available yet, so we don't know what machine type we'd want to use, and QEMU isn't even installed on the hosts doing this decision making. And libvirt is installed? Make that host query utility separate from qemu then, make libvirt depend on it. Currently OpenStack just has to pretend that CPU models don't change based on machine type. Most of the time we'll be lucky and that won't hurt us, but obviously it is not a desirable thing to have todo. Still wonder whether if it's a theoretical problem. If yes, why write a ton of code to support it? On the other hand (2) is a pretty minimal change to QEMU and is more flexible than (1) because it allows CPU model versions to be decoupled from machine types (but this was already discussed a lot in the other emails in this thread). Jirka I'm fine with the change itself, it's useful e.g.
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 11:23:16PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:42:04PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? Huh, I never said we wanted to bring back bugs. This is about allowing libvirt to fix the CPU bugs in a way that is independant of the machine types and portable across hypervisors we deal with. We're absolutely still going to fix CPU model bugs and ensure stable guest ABI. Regards, Daniel So any single CPU flag now needs to be added in - kvm - qemu - libvirt Next thing libvirt will decide it's a policy thing and so needs to be pushed up to openstack. I don't think that will happen, but if they really decide
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On 24/06/2015 16:54, Peter Maydell wrote: It's certainly okay for libvirt and OpenStack to use the host CPU features in order to check whether a node will run a given VM. However, libvirt should trust that QEMU developers will not prevent a VM from running on a previously viable host, just because you change the machine type. Note that this might be true for x86 but isn't (in full generality) necessarily so for other architectures like ARM. For instance if you have a host machine with a GICv3 with no v2-back-compatibility support, then you can run a VM whose machine type specifies a GICv3, but not machine types that require a GICv2... Hmm, that was too terse. Just because you change the machine type *to a newer version* with the same chipset. Paolo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On 24 June 2015 at 15:38, Paolo Bonzini pbonz...@redhat.com wrote: It's certainly okay for libvirt and OpenStack to use the host CPU features in order to check whether a node will run a given VM. However, libvirt should trust that QEMU developers will not prevent a VM from running on a previously viable host, just because you change the machine type. Note that this might be true for x86 but isn't (in full generality) necessarily so for other architectures like ARM. For instance if you have a host machine with a GICv3 with no v2-back-compatibility support, then you can run a VM whose machine type specifies a GICv3, but not machine types that require a GICv2... thanks -- PMM
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On 24/06/2015 16:16, Eduardo Habkost wrote: So any single CPU flag now needs to be added in - kvm - qemu - libvirt Next thing libvirt will decide it's a policy thing and so needs to be pushed up to openstack. I don't think that will happen, but if they really decide do do it, why should we try to stop them? libvirt and OpenStack know what their users do/need better than us, and if they believe moving data to OpenStack will provide what users need, they are free to do it. I trust libvirt developers to do the right thing, here. After talking to Daniel for more than 1 hour I actually think that OpenStack's scheduler is totally broken with respect to QEMU upgrades. For some reason they focused on CPU model changes, but actually there's no past case where runnability actually changed with a QEMU upgrade, and there will be no such future case; even without enforce, QEMU introduces new models long after all features have been available in KVM. At this point CPU is no different from any other device. At the same time, OpenStack's scheduler is not trying to use a machine type that is available on all nodes of a compute node pool. I'm not sure how one can be sure that migration would succeed in these circumstances. It's certainly okay for libvirt and OpenStack to use the host CPU features in order to check whether a node will run a given VM. However, libvirt should trust that QEMU developers will not prevent a VM from running on a previously viable host, just because you change the machine type. And OpenStack should _really_ use the machine type as the abstract representation of what is compatible with what inside a node pool. This is what RHEV does, and I see no reason to do it differently. In particular there will not be excessive fragmentation of the nodes into too many pools, because a deployment with too many active machine types is just asking for trouble. The possible exception are weird cases involving nested virt and outdated QEMU on the L0 host, but even those cases can be fixed just by having an up-to-date cpu_map.xml. Other random notes from my chat: 1) libvirt should _not_ change the flags that the user passes via XML just because it thinks that QEMU has those flags. This makes it possible for libvirt to keep cpu_map.xml up-to-date without worrying about versioning. 2) libvirt should not add/remove flags when the user specifies host-model (i.e. -cpu SandyBridge, not -cpu SandyBridge,+f16c,+rdrand,+erms,+whatever). host-model has had so many bugs reported for it that I hope this could be done unconditionally even if it is not backwards-compatible. Or perhaps introduce a new name and deprecate host-model. I don't know. 3) regarding enforce, there are indeed some cases where it would break: - Haswell/Broadwell CPU model after TSX removal - qemu64 with KVM - pretty much everything including qemu64 with TCG So libvirt here could allow _now_ the user to specify enforce, either via XML or via qemu.conf (or via XML + a default specified via qemu.conf). So, I _hate_ to block a feature that is anyway useful for debugging. And this feels too much like the typical kernel developers don't like systemd rant. But it looks like this feature is not only too easy to misuse: there are _already_ plans for misusing it and put the whole machine compatibility issue under the rug in OpenStack. So I agree with Andreas, and would prefer to have a serious discussion with the OpenStack folks before accepting it. Paolo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Wed, Jun 24, 2015 at 11:16:51AM -0300, Eduardo Habkost wrote: IMHO -M pc is not supposed to mean can break at any time. It means it may have new host-side requirements and may become runnable in your host (or require additional command-line flags to run) at any time. That would be pretty bad. I don't think we ever had such cases in practice. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 11:28:06PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 02:42:37PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 07:29:15PM +0200, Andreas Färber wrote: In summary you seem to be saying that all the years we have spent fiddling around with those mind-boggling compat_props in QEMU were in vain because libvirt now wants to start their own versioning system to give users more degrees of freedom even when you can't articulate a single concrete reason why users may want to do so. I had a similar reaction when I learned about this libvirt expectation/requirement I was never aware of. But we spent lots of effort trying to do things differently doesn't seem like a valid justification for design decision. Users will be hurt because they'll run untested configurations seems like a valid reason. I trust libvirt developers to test their CPU definitions as carefully as we test ours. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 11:34:45PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 02:11:22PM -0300, Eduardo Habkost wrote: Even if it is a bug fix. If it is a change that can make the VM unrunnable, it needs to be controlled by a separate flag, not by the machine-type. I agree - command line compatibility is important. But we are supposed to provide that. I am surprised that libvirt suddenly wants to avoid some command line flags because they are not stable. IMHO we did something wrong here if so. Maybe there was a valid reason for it. But then won't it apply to libvirt as well? Maybe we are having the same misunderstanding here: the problem is not compatibility/stability of existing machines, but the kind of (intentional) changes introduced in _new_ machines (when the -machine argument is changed). There are two kinds of changes introduced in new machines: 1) Guest-side-only ABI changes: those are OK, libvirt normally ignore them, they can't make a VM not-runnable. 2) Changes in the host-side dependencies: those need to be more carefully controlled by libvirt. That's where CPU features are special: all CPU features depend KVM-side features, and enabling them by default on new machines makes it impossible for libvirt to know/report in advance what's necessary to make a VM runnable and to implement their existing runnability APIs[1]. Unless we guarantee that QEMU would never introduce type-(2) changes in new machines (which I don't think will ever happen because that means never changing existing CPU models in QEMU), libvirt needs to control CPU features individually (that's why they need -cpu custom). Now, if people want to update CPU models outside the QEMU binary, that might be doable simply by moving them to a separate package, with a text file that QEMU reads at startup. You seem to be describing exactly what is made possible by -cpu custom -readconfig cpu-config-file [1] http://libvirt.org/html/libvirt-libvirt-host.html#virConnectCompareCPU (See the cover letter of this series) -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Wed, Jun 24, 2015 at 11:18:18AM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 11:28:06PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 02:42:37PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 07:29:15PM +0200, Andreas Färber wrote: In summary you seem to be saying that all the years we have spent fiddling around with those mind-boggling compat_props in QEMU were in vain because libvirt now wants to start their own versioning system to give users more degrees of freedom even when you can't articulate a single concrete reason why users may want to do so. I had a similar reaction when I learned about this libvirt expectation/requirement I was never aware of. But we spent lots of effort trying to do things differently doesn't seem like a valid justification for design decision. Users will be hurt because they'll run untested configurations seems like a valid reason. I trust libvirt developers to test their CPU definitions as carefully as we test ours. Maybe someone can do a write-up explaining how do requirements and needs differ? So far, it looks like gratituos code duplication based on a mis-understanding, or highly unlikely theoretical what-if scenarious. Basing our stable interfaces on such grounds might not be a good idea. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 24.06.2015 um 16:57 schrieb Michael S. Tsirkin: On Wed, Jun 24, 2015 at 04:35:13PM +0200, Andreas Färber wrote: Am 24.06.2015 um 16:19 schrieb Michael S. Tsirkin: On Wed, Jun 24, 2015 at 11:16:51AM -0300, Eduardo Habkost wrote: IMHO -M pc is not supposed to mean can break at any time. It means it may have new host-side requirements and may become runnable in your host (or require additional command-line flags to run) at any time. That would be pretty bad. I don't think we ever had such cases in practice. Why is that bad or unexpected? If you install new software, it may have new dependencies. QEMU would be no different from other software there. That's just basic compatibility. If I run using same flags, I expect compatible behaviour. How would you like it if each time you update bash, all your scripts had to stop working, unless you specify a specific compability flag in scripts, in which case you miss (some) bugfixes? Or if you found code snippets and they can't both work. Why? Because snippet 1 says request version 2.4 compatibility and another says use version 2.5 compatibility, so you can't use both. Each time you break command line compatibility, you invalidate an unknown chunk of useful documentation somewhere. Not sure if we're talking about the same thing. For starters, bash or qemu-system-foo may have a dependency on a new library. sudo recently grew dependencies on some auditing syscall. Certain machine or CPU level KVM features may require new ioctls. Therefore new features may have new dependencies, and by default when not in backwards-compatibility mode we want to enable such new features and not just some ancient frozen subset - which would be comparable to Solaris' /bin/sh being a really ancient version with new ones under new names. Don't understand what you mean with those two snippets. Either way it seems a fundamental non-issue at the moment. qemu64 apart from x2apic did not change much. I certainly didn't propose any functional feature changes myself and only took patches once okayed from KVM and x86 reviewers, raising the compat_props card. However, looking beyond the artificial qemu64, a causal problem beneath this discussion seems to be our white-listing of CPU features rather than transparently black-listing all but implemented features. Apart from omission bugs, there's two ways CPU features can get added, implementation in TCG and kernel support in KVM. Only the KVM support depends on the host, whereas the TCG support solely depends on the QEMU version. Bug fixes would not fall into the implementation-dependent category. And for the feature-moratorium Paolo proposed I wonder whether we have any mechanism to assure and test that? Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Wed, Jun 24, 2015 at 04:38:51PM +0200, Paolo Bonzini wrote: [...] 1) libvirt should _not_ change the flags that the user passes via XML just because it thinks that QEMU has those flags. This makes it possible for libvirt to keep cpu_map.xml up-to-date without worrying about versioning. That's true, and it would solve cases like people asking libvirt for Haswell without rdand and getting QEMU running -cpu Haswell (which has rdrand enabled) because libvirt thinks Haswell doesn't have rdrand. 2) libvirt should not add/remove flags when the user specifies host-model (i.e. -cpu SandyBridge, not -cpu SandyBridge,+f16c,+rdrand,+erms,+whatever). host-model has had so many bugs reported for it that I hope this could be done unconditionally even if it is not backwards-compatible. Or perhaps introduce a new name and deprecate host-model. I don't know. In other words, libvirt should not assume anything about the QEMU-side CPU models. If the user asks for Broadwell, it should use -cpu Broadwell, if the user asks for Broadwell + fpu, it should use -cpu Broadwell,+fpu even if it believes every CPU model since 486 has FPU enabled. The reason for that is that we may still introduce CPU model changes, as long as they are not going to mean new host-side dependencies (see commit 78a611f1936b3eac8ed78a2be2146a742a85212c for an example, where we added f16c and rdand to Haswell and Broadwell). 3) regarding enforce, there are indeed some cases where it would break: - Haswell/Broadwell CPU model after TSX removal - qemu64 with KVM - pretty much everything including qemu64 with TCG - Everything involving KVM CPUID features, that change depending on the kernel version. - Intel CPU models that had VMX enabled in older machine-types, and may or may not have VMX enabled depending on kernel VMX nesting configuration. So libvirt here could allow _now_ the user to specify enforce, either via XML or via qemu.conf (or via XML + a default specified via qemu.conf). More precisely, libvirt would emulate enforce mode by checking the filtered-features, because enforce error messages are not machine-friendly. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Wed, Jun 24, 2015 at 11:24:46AM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 11:34:45PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 02:11:22PM -0300, Eduardo Habkost wrote: Even if it is a bug fix. If it is a change that can make the VM unrunnable, it needs to be controlled by a separate flag, not by the machine-type. I agree - command line compatibility is important. But we are supposed to provide that. I am surprised that libvirt suddenly wants to avoid some command line flags because they are not stable. IMHO we did something wrong here if so. Maybe there was a valid reason for it. But then won't it apply to libvirt as well? Maybe we are having the same misunderstanding here: the problem is not compatibility/stability of existing machines, but the kind of (intentional) changes introduced in _new_ machines (when the -machine argument is changed). There are two kinds of changes introduced in new machines: 1) Guest-side-only ABI changes: those are OK, libvirt normally ignore them, they can't make a VM not-runnable. 2) Changes in the host-side dependencies: those need to be more carefully controlled by libvirt. That's where CPU features are special: all CPU features depend KVM-side features, and enabling them by default on new machines makes it impossible for libvirt to know/report in advance what's necessary to make a VM runnable and to implement their existing runnability APIs[1]. Not all that special. e.g. virtio device offloads depend on host networking's ability to offload them too. And hey, QEMU is part of host too, so if you want to be pedantic, any device configuration might be broken on an old host and fixed on a new one, so vm seems to run but will lose data or crashes. All this is unusual enough that no one bothers. Unless we guarantee that QEMU would never introduce type-(2) changes in new machines (which I don't think will ever happen because that means never changing existing CPU models in QEMU), libvirt needs to control CPU features individually (that's why they need -cpu custom). Did we ever do it? I don't think so, so yes, we very likely never will. It's as if someone wrote a wrapper around all kernel system calls on the assumption that kernel can not guarantee kernel/userspace ABI will never change. It can and it does. So let's promise not to break things, and avoid a ton of copy and paste bugs. Now, if people want to update CPU models outside the QEMU binary, that might be doable simply by moving them to a separate package, with a text file that QEMU reads at startup. You seem to be describing exactly what is made possible by -cpu custom -readconfig cpu-config-file [1] http://libvirt.org/html/libvirt-libvirt-host.html#virConnectCompareCPU (See the cover letter of this series) Only if you also add the config files as part of qemu, reworking existing models to load config file from a pre-defined directory. My whole point is to avoid useless code duplication. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Wed, Jun 24, 2015 at 04:35:13PM +0200, Andreas Färber wrote: Am 24.06.2015 um 16:19 schrieb Michael S. Tsirkin: On Wed, Jun 24, 2015 at 11:16:51AM -0300, Eduardo Habkost wrote: IMHO -M pc is not supposed to mean can break at any time. It means it may have new host-side requirements and may become runnable in your host (or require additional command-line flags to run) at any time. That would be pretty bad. I don't think we ever had such cases in practice. Why is that bad or unexpected? If you install new software, it may have new dependencies. QEMU would be no different from other software there. That's just basic compatibility. If I run using same flags, I expect compatible behaviour. How would you like it if each time you update bash, all your scripts had to stop working, unless you specify a specific compability flag in scripts, in which case you miss (some) bugfixes? Or if you found code snippets and they can't both work. Why? Because snippet 1 says request version 2.4 compatibility and another says use version 2.5 compatibility, so you can't use both. Each time you break command line compatibility, you invalidate an unknown chunk of useful documentation somewhere. Yesterday Eduardo said it was about having a fixed version installed and therein switching from a legacy machine to a newer machine. In that case the dependencies existed ever since installation but were not immediately visible to the end user due to our stability guarantees. Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 24.06.2015 um 16:19 schrieb Michael S. Tsirkin: On Wed, Jun 24, 2015 at 11:16:51AM -0300, Eduardo Habkost wrote: IMHO -M pc is not supposed to mean can break at any time. It means it may have new host-side requirements and may become runnable in your host (or require additional command-line flags to run) at any time. That would be pretty bad. I don't think we ever had such cases in practice. Why is that bad or unexpected? If you install new software, it may have new dependencies. QEMU would be no different from other software there. Yesterday Eduardo said it was about having a fixed version installed and therein switching from a legacy machine to a newer machine. In that case the dependencies existed ever since installation but were not immediately visible to the end user due to our stability guarantees. Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On 24/06/2015 17:58, Eduardo Habkost wrote: 3) regarding enforce, there are indeed some cases where it would break: - Haswell/Broadwell CPU model after TSX removal - qemu64 with KVM - pretty much everything including qemu64 with TCG - Everything involving KVM CPUID features, that change depending on the kernel version. - Intel CPU models that had VMX enabled in older machine-types, and may or may not have VMX enabled depending on kernel VMX nesting configuration. So libvirt here could allow _now_ the user to specify enforce, either via XML or via qemu.conf (or via XML + a default specified via qemu.conf). More precisely, libvirt would emulate enforce mode by checking the filtered-features, because enforce error messages are not machine-friendly. Even better, because it would let libvirt special-case stuff like TSX, VMX and SVM. Paolo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. Expanding on that, but tieing the CPU model to the machine type, QEMU has in turn effectively tied the machine type to the host hardware. eg, switching to a newer machine type, may then prevent the guest from being able to launch on the hardware that it was previously able to run on, due to some new requirement of the CPU model associated with the machine type. Libvirt wants the CPU models to be independant of the machine type, so in general only the CPU model is dependant on hardware capabilities and machine type is isolated from hardware. Libvirt still intends to do versioning of the CPU models, but the versioning will be separate from the versioning of the machine types, and will be handled by libvirt itself. This also allows us to get further towards our goal which is to have a consistent representation of CPU models across all libvirt hypervisors. eg the same libvirt CPU model and versions can be made consistent across kvm, xen, vmware, etc, as they're not longer changing behind our back based on the qemu machine type. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. Regards, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? Huh, I never said we wanted to bring back bugs. This is about allowing libvirt to fix the CPU bugs in a way that is independant of the machine types and portable across hypervisors we deal with. We're absolutely still going to fix CPU model bugs and ensure stable guest ABI. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 18:32 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Uh? I don't remember anybody suggesting changing CPU models on existing machines. We always tried to keep existing machines compatible. Yes, we try in general. And in a few cases I was overruled, possibly related to TCG feature filtering or something. Thought that was the problem here - apparently not. Explanations seem to be the culprit here! BTW your list does not answer my question. You would need examples where a CPU model changes between machines, and I am not aware of any example beyond the intentional -x.y variations. There are differences between KVM and TCG though, did you mean that? i440fx and q35 should be identical and isa-pc, too, and none anyway. None of this has anything to do with the host CPU. We are talking about the -x.y variations (that, yes, are intentional). But the fact that CPU features change (even the intentional ones in the -x.y machine variations) affect runnability of VMs (because enabling new CPU features in KVM require it to be supported by the host kernel code and by the host CPU). I was not thinking about the KVM and TCG differences, but this may also help libvirt deal with the KVM and TCG differences if necessary. I don't know what you mean by i440fx and q35 should be identical above. In this thread there was a claim that CPU models varied between machine types. I am saying that there should be no CPU model differences between pc-i440fx-2.3 and pc-q35-2.3 etc. Thus the CPU model is not tied to one machine, but to the version of QEMU, with -x.y matching the corresponding release. No news to you, I would hope? Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 19:08 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:44:57PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:38 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? I didn't take the time to count them, but I bet most of the commits I listed on my previous e-mail message are not bug fixes, but new features. Huh? Of course the latest machine model get new features. The point is that the previous ones don't and that's what we are providing them for - libvirt is expected to choose one machine and the contract with QEMU is that for that machine the CPU does *not* grow new features, and we're going at great lengths to achieve that. So this thread feels more and more weird... We are not talking about changes to existing machines. We are talking about having changes introduced in new machines (the one we did on purpose) affecting the runnability of the VM. You are talking abstract! Example 1: Point A: Machine pc-i440fx-2.3 exists Runs or runs not. Point B: Machine pc-i440fx-2.3 still exists Still runs or runs not due to guest ABI stability rules. Example 2: Point A: pc-i440fx-2.4 does not exist in 2.3 Does not run becomes it doesn't exist. Point B: New pc-i440fx-2.4 Runs or does not run, and if so has more features than pc-i440fx-2.3. There is no runnability problem - either it runs or it doesn't, but there's no change over time. This is what the machine -x.y versioning is all about. Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 07:10:11PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:53 schrieb Daniel P. Berrange: On Tue, Jun 23, 2015 at 06:40:19PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:25 schrieb Daniel P. Berrange: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. This coupling is by design and we expect all KVM/QEMU users to adhere to it, including those that use the libvirt tool (which I assume is going to be the majority of KVM users). Either you want a certain backwards-compatible machine and CPU, or you want the latest and greatest - why in the world mix and match?! As mentioned, changes/fixes to the CPU model can affect the ability to launch the guest on a particular host, so we need the ability to control when those CPU changes are activated for a guest, separately from the machine type. Why? Today's libvirt with QEMU 2.3 resolves pc machine to pc-i440fx-2.3 and the guest XML stays that way. When we add new features for 2.4, 2.3 is guaranteed to stay compatible. Any change would involve the libvirt user actively switching from pc-i440fx-2.3 to a different machine such as upcoming pc-i440fx-2.4. Why do you need to change the CPU separately? Why would a user want to run 2.2's CPU with a 2.3 machine? Or a 2.3 machine with a 2.4 CPU? That's nonsense. If you want to tweak features, you already have command line options available to do so on the basis of what the selected machine provides. Because pure guest-side ABI changes are different from changes that also have additional host-side requirements, so we want to untie both things. About being able to tweak features today, that's true: we have command-line options for most stuff and that's _almost_ enough for what libvirt needs. What's missing is something to avoid silently getting new features that libvirt aren't aware of (but may make the VM unrunnable). The purpose of -cpu custom is to ensure no new host side feature requirement will be introduced silently, even if choosing a different machine. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 02:24:16PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 07:10:11PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:53 schrieb Daniel P. Berrange: On Tue, Jun 23, 2015 at 06:40:19PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:25 schrieb Daniel P. Berrange: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. This coupling is by design and we expect all KVM/QEMU users to adhere to it, including those that use the libvirt tool (which I assume is going to be the majority of KVM users). Either you want a certain backwards-compatible machine and CPU, or you want the latest and greatest - why in the world mix and match?! As mentioned, changes/fixes to the CPU model can affect the ability to launch the guest on a particular host, so we need the ability to control when those CPU changes are activated for a guest, separately from the machine type. Why? Today's libvirt with QEMU 2.3 resolves pc machine to pc-i440fx-2.3 and the guest XML stays that way. When we add new features for 2.4, 2.3 is guaranteed to stay compatible. Any change would involve the libvirt user actively switching from pc-i440fx-2.3 to a different machine such as upcoming pc-i440fx-2.4. Why do you need to change the CPU separately? Why would a user want to run 2.2's CPU with a 2.3 machine? Or a 2.3 machine with a 2.4 CPU? That's nonsense. If you want to tweak features, you already have command line options available to do so on the basis of what the selected machine provides. Because pure guest-side ABI changes are different from changes that also have additional host-side requirements, so we want to untie both things. About being able to tweak features today, that's true: we have command-line options for most stuff and that's _almost_ enough for what libvirt needs. What's missing is something to avoid silently getting new features that libvirt aren't aware of (but may make the VM unrunnable). The purpose of -cpu custom is to ensure no new host side feature requirement will be introduced silently, even if choosing a different machine. It also has a positive impact on maintenance. By allowing libvirt to control the CPU definitions, in cases where there is a need to push out a CPU model bugfix, libvirt will often be able to make the update available to users via a new CPU model version, without them having to do a lock-step upgrade to QEMU at the same time. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: commit 726a8ff68677d8d5fba17eb0ffb85076bfb598dc Author: Eduardo Habkost ehabk...@redhat.com Date: Fri Apr 10 14:45:00 2015 -0300 target-i386: Remove AMD feature flag aliases from CPU model table When CPU vendor is AMD, the AMD feature alias bits on CPUID[0x8001].EDX are already automatically copied from CPUID[1].EDX on x86_cpu_realizefn(). When CPU vendor is Intel, those bits are reserved and should be zero. On either case, those bits shouldn't be set in the CPU model table. Reviewed-by: Igor Mammedov imamm...@redhat.com Signed-off-by: Eduardo Habkost ehabk...@redhat.com commit 13704e4c455770d500d6b87b117e32f0d01252c9 Author: Eduardo Habkost ehabk...@redhat.com Date: Thu Jan 22 17:22:54 2015 -0200 target-i386: Disable HLE and RTM on Haswell Broadwell All Haswell CPUs and some Broadwell CPUs were updated by Intel to have the HLE and RTM features disabled. This will prevent -cpu Haswell,enforce and -cpu Broadwell,enforce from running out of the box on those CPUs. Disable those features by default on Broadwell and Haswell CPU models, starting on pc-*-2.3. Users who want to use those features can enable them explicitly on the command-line. Signed-off-by: Eduardo Habkost ehabk...@redhat.com Signed-off-by: Paolo Bonzini pbonz...@redhat.com commit 78a611f1936b3eac8ed78a2be2146a742a85212c Author: Paolo Bonzini pbonz...@redhat.com Date: Fri Dec 5 10:52:46 2014 +0100 target-i386: add f16c and rdrand to Haswell and Broadwell Both were added in Ivy Bridge (for which we do not have a CPU model yet!). Reviewed-by: Eduardo Habkost ehabk...@redhat.com Signed-off-by: Paolo Bonzini pbonz...@redhat.com commit b3a4f0b1a072a467d003755ca0e55c5be38387cb Author: Paolo Bonzini pbonz...@redhat.com Date: Wed Dec 10 14:12:41 2014 -0200 target-i386: add VME to all CPUs vm86 mode extensions date back to the 486. All models should have them. Signed-off-by: Paolo Bonzini pbonz...@redhat.com Signed-off-by: Eduardo Habkost ehabk...@redhat.com Signed-off-by: Paolo Bonzini pbonz...@redhat.com commit 0bb0b2d2fe7f645ddaf1f0ff40ac669c9feb4aa1 Author: Paolo Bonzini pbonz...@redhat.com Date: Mon Nov 24 15:54:43 2014 +0100 target-i386: add feature flags for CPUID[EAX=0xd,ECX=1] These represent xsave-related capabilities of the processor, and KVM may or may not support them. Add feature
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 04:51:00PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. Expanding on that, but tieing the CPU model to the machine type, QEMU has in turn effectively tied the machine type to the host hardware. eg, switching to a newer machine type, may then prevent the guest from being able to launch on the hardware that it was previously able to run on, due to some new requirement of the CPU model associated with the machine type. So why not keep machine type stable? Libvirt wants the CPU models to be independant of the machine type, so in general only the CPU model is dependant on hardware capabilities and machine type is isolated from hardware. Libvirt still intends to do versioning of the CPU models, but the versioning will be separate from the versioning of the machine types, and will be handled by libvirt itself. This also allows us to get further towards our goal which is to have a consistent representation of CPU models across all libvirt hypervisors. eg the same libvirt CPU model and versions can be made consistent across kvm, xen, vmware, etc, as they're not longer changing behind our back based on the qemu machine type. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 18:42 schrieb Daniel P. Berrange: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? Huh, I never said we wanted to bring back bugs. This is about allowing libvirt to fix the CPU bugs in a way that is independant of the machine types and portable across hypervisors we deal with. We're absolutely still going to fix CPU model bugs and ensure stable guest ABI. No, that's contradictory! Through the -x.y machines we leave bugs in the old models *exactly* to assure a stable guest ABI. Fixes are only be applied to new machines, thus I'm pointing out that you should not use a new CPU model with an old machine type. Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 06:47:16PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:42 schrieb Daniel P. Berrange: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? Huh, I never said we wanted to bring back bugs. This is about allowing libvirt to fix the CPU bugs in a way that is independant of the machine types and portable across hypervisors we deal with. We're absolutely still going to fix CPU model bugs and ensure stable guest ABI. No, that's contradictory! Through the -x.y machines we leave bugs in the old models *exactly* to assure a stable guest ABI. Fixes are only be applied to new machines, thus I'm pointing out that you should not use a new CPU model with an old machine type. I'm not saying that libvirt would ever allow a silent guest ABI change. Given a libvirt XML config, the guest ABI will never change without an explicit action on the part of the app/user to change the XML. This is all about dealing with the case where the app / user conciously needs/wants to opt-in to a guest ABI change for the guest. eg they wish to make use of some bug fix or feature improvement in the new machine type, but they do *not* wish to have the CPU model changed, because of some CPU model change that is incompatible with their hosts' CPUs. Conversely, they may wish to get access to a new CPU model, but not wish to have the rest of the guest ABI change. In both cases the user is explicitly opt-ing in the ABI change with knowledge about what this might mean for the guest OS. Currently we are tieing users hands by forcing CPU and machine types to change in lockstep. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 07:29:15PM +0200, Andreas Färber wrote: In summary you seem to be saying that all the years we have spent fiddling around with those mind-boggling compat_props in QEMU were in vain because libvirt now wants to start their own versioning system to give users more degrees of freedom even when you can't articulate a single concrete reason why users may want to do so. I had a similar reaction when I learned about this libvirt expectation/requirement I was never aware of. But we spent lots of effort trying to do things differently doesn't seem like a valid justification for design decision. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 18:25 schrieb Daniel P. Berrange: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. This coupling is by design and we expect all KVM/QEMU users to adhere to it, including those that use the libvirt tool (which I assume is going to be the majority of KVM users). Either you want a certain backwards-compatible machine and CPU, or you want the latest and greatest - why in the world mix and match?! Would a qemu64-2.3 model help here that pc*-2.3 could use? I believe that's been proposed in the past. I don't oppose the idea of a fully-custom CPU, but this blatant attempt of ignoring QEMU's CPU versioning by libvirt worries me. Regards, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 06:47:16PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:42 schrieb Daniel P. Berrange: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? Huh, I never said we wanted to bring back bugs. This is about allowing libvirt to fix the CPU bugs in a way that is independant of the machine types and portable across hypervisors we deal with. We're absolutely still going to fix CPU model bugs and ensure stable guest ABI. No, that's contradictory! Through the -x.y machines we leave bugs in the old models *exactly* to assure a stable guest ABI. Fixes are only be applied to new machines, thus I'm pointing out that you should not use a new CPU model with an old machine type. They don't need to use a new model with an old machine-type (although I don't see a reason to prevent that). They need to be able to change the machine-type to a new one without getting any changes that would make the VM not runnable in the same host. Even if it is a bug fix. If it is a change that can make the VM unrunnable, it needs to be controlled by a separate flag, not by the machine-type. --
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 07:18:06PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:08 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:44:57PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:38 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? I didn't take the time to count them, but I bet most of the commits I listed on my previous e-mail message are not bug fixes, but new features. Huh? Of course the latest machine model get new features. The point is that the previous ones don't and that's what we are providing them for - libvirt is expected to choose one machine and the contract with QEMU is that for that machine the CPU does *not* grow new features, and we're going at great lengths to achieve that. So this thread feels more and more weird... We are not talking about changes to existing machines. We are talking about having changes introduced in new machines (the one we did on purpose) affecting the runnability of the VM. You are talking abstract! I am just talking about a different problem, and I don't know if you are purposely trying to ignore it, or are just denying that it is a problem. Example 1: Point A: Machine pc-i440fx-2.3 exists Runs or runs not. Point B: Machine pc-i440fx-2.3 still exists Still runs or runs not due to guest ABI stability rules. If you didn't change the machine name, this is not the problem we are talking about. Example 2: Point A: pc-i440fx-2.4 does not exist in 2.3 Does not run becomes it doesn't exist. Point B: New pc-i440fx-2.4 Runs or does not run, and if so has more features than pc-i440fx-2.3. If you didn't change the machine name, this is not the problem we are talking about. There is no runnability problem - either it runs or it doesn't, but there's no change over time. This is what the machine -x.y versioning is all about. Let's try a concrete example: * User is running a kernel that can't emulate x2apic * User is running pc-i440fx-1.7 * User wants the gigabyte alignment change implemented by commit bb43d3839c29b17a2f5c122114cd4ca978065a18 * User changes machine to pc-i440fx-2.0 * x2apic is now enabled by default in all CPU models * VM with the same configuration (just the machine change) is not runnable anymore in the same host -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 19:27 schrieb Daniel P. Berrange: On Tue, Jun 23, 2015 at 07:18:06PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:08 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:44:57PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:38 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? I didn't take the time to count them, but I bet most of the commits I listed on my previous e-mail message are not bug fixes, but new features. Huh? Of course the latest machine model get new features. The point is that the previous ones don't and that's what we are providing them for - libvirt is expected to choose one machine and the contract with QEMU is that for that machine the CPU does *not* grow new features, and we're going at great lengths to achieve that. So this thread feels more and more weird... We are not talking about changes to existing machines. We are talking about having changes introduced in new machines (the one we did on purpose) affecting the runnability of the VM. You are talking abstract! Example 1: Point A: Machine pc-i440fx-2.3 exists Runs or runs not. Point B: Machine pc-i440fx-2.3 still exists Still runs or runs not due to guest ABI stability rules. Example 2: Point A: pc-i440fx-2.4 does not exist in 2.3 Does not run becomes it doesn't exist. Point B: New pc-i440fx-2.4 Runs or does not run, and if so has more features than pc-i440fx-2.3. There is no runnability problem - either it runs or it doesn't, but there's no change over time. This is what the machine -x.y versioning is all about. Consider a host currently running QEMU 2.3 with machine type pc-i440fx-2.3 used with SandyBridge. Now consider the def of SandyBridge was buggy and so in QEMU 2.4 we add the missing CPU feature flag 'wizz', and only enable that new feature flag with pc-i440fx-2.4 Now consider there was a bug in the virtio-scsi driver that we also fixed in QEMU 2.4 and thus pc-i440fx-2.4 includes that fix. Updating from pc-i440fx-2.3 to pc-i440fx-2.4 has a dependancy on the host CPU including the 'wizz' flag that was added. This new CPU feature can prevent the user from using the new machine type to get the virtio-scsi bug fix. Thanks for this example! :) Only if you use -cpu ...,enforce, no? The KVM feature filtering should take care of dropping features that are not available otherwise. So we seem to be getting to the interesting case of the same machine (different from what was said previously!) but different hosts. The QOM property gives you insights into which feature bits are set for the machine for the model (and for s390x I saw QMP extensions to the same effect, I thought). That way you could discover features to disable. However you'll only ever know which ones work once you've tried it once, right? I'm pretty sure that we've had discussions with Anthony and Avi on the same topic ages ago. Alex also made the -cpu best proposal long ago. In general, if you want to run on a group of hosts, then you need to figure out a common denominator - a CPU model name and optionally features to enable/disable. If that is the whole problem here, then why not just add a global flag to only enable explicitly requested KVM features? All other features should not depend on the host, and the whole discussion about -x.y seems like a distraction. Regards, Andreas Separately versioning CPU models still lets us preserve a stable guest ABI by default, but allows more flexibility when choosing to opt out of the ABI for a particular upgrade. Regards, Daniel -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 05:00:54PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 05:56:35PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 04:51:00PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. Expanding on that, but tieing the CPU model to the machine type, QEMU has in turn effectively tied the machine type to the host hardware. eg, switching to a newer machine type, may then prevent the guest from being able to launch on the hardware that it was previously able to run on, due to some new requirement of the CPU model associated with the machine type. So why not keep machine type stable? There are many reasons to choose a particular machine type - for example, to achieve migration compat between hosts with different QEMU versions, This might make you use an old machine type. It will never make you use a newer machine type so you will never run into problems. or to enable access to some performance or bug fix in the machine type in question. Performance/bugfixes is exactly why we change these though. Users / apps need to be free to make those decisions, without being restricted by changes in the CPU model which may affect what hardware the machine type can be used on. The current use of machine types for CPU model versioning is placing users between a rock hard place, giving them impossible decisions about which bad behaviour/bug they're willing to accept. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. Regards, Daniel But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? I didn't take the time to count them, but I bet most of the commits I listed on my previous e-mail message are not bug fixes, but new features. Also, it doesn't matter if the change is a bug fix or a new feature: if it affects runnability of the VM, it has more consequences than a simple guest-side ABI change, and libvirt can't tie it to the machine-type so it needs another flag to enable it. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 18:38 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? I didn't take the time to count them, but I bet most of the commits I listed on my previous e-mail message are not bug fixes, but new features. Huh? Of course the latest machine model get new features. The point is that the previous ones don't and that's what we are providing them for - libvirt is expected to choose one machine and the contract with QEMU is that for that machine the CPU does *not* grow new features, and we're going at great lengths to achieve that. So this thread feels more and more weird... Regards, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 07:41:50PM +0200, Andreas Färber wrote: [...] If that is the whole problem here, then why not just add a global flag to only enable explicitly requested KVM features? All other features should not depend on the host, and the whole discussion about -x.y seems like a distraction. Now replace KVM features with CPU fatures, because all CPU features are KVM features, as all of them depend on KVM code enabling them on GET_SUPPORTED_CPUID. Thus, the global flag to only enable explicitly request KVM features on CPUs is -cpu custom, which doesn't enable any CPU feature at all. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 06:44:57PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:38 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? I didn't take the time to count them, but I bet most of the commits I listed on my previous e-mail message are not bug fixes, but new features. Huh? Of course the latest machine model get new features. The point is that the previous ones don't and that's what we are providing them for - libvirt is expected to choose one machine and the contract with QEMU is that for that machine the CPU does *not* grow new features, and we're going at great lengths to achieve that. So this thread feels more and more weird... We are not talking about changes to existing machines. We are talking about having changes introduced in new machines (the one we did on purpose) affecting the runnability of the VM. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 19:13 schrieb Daniel P. Berrange: On Tue, Jun 23, 2015 at 06:47:16PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:42 schrieb Daniel P. Berrange: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? Huh, I never said we wanted to bring back bugs. This is about allowing libvirt to fix the CPU bugs in a way that is independant of the machine types and portable across hypervisors we deal with. We're absolutely still going to fix CPU model bugs and ensure stable guest ABI. No, that's contradictory! Through the -x.y machines we leave bugs in the old models *exactly* to assure a stable guest ABI. Fixes are only be applied to new machines, thus I'm pointing out that you should not use a new CPU model with an old machine type. I'm not saying that libvirt would ever allow a silent guest ABI change. Given a libvirt XML config, the guest ABI will never change without an explicit action on the part of the app/user to change the XML. This is all about dealing with the case where the app / user conciously needs/wants to opt-in to a guest ABI change for the guest. eg they wish to make use of some bug fix or feature improvement in the new machine type, but they do *not* wish to have the CPU model changed, because of some CPU model change that is incompatible with their hosts' CPUs. Conversely, they may wish to get access to a new CPU model, but not wish to have the rest of the guest ABI change. In both cases the user is explicitly opt-ing in the ABI change with knowledge about what this might mean for the guest OS. Currently we are tieing users hands by forcing CPU and machine types to change in lockstep. Look, if you keep repeating that users may wish to do random combinations if we give them the ability to do so, that is not helping me understand. Right now they can't. If you don't have a single use case of what CPU model, KVM feature, CPUID bit and guest may constitute a valid reason to do so, I disagree with adding such a crazy feature. In summary you seem to be saying that all the years we have spent fiddling around with those mind-boggling compat_props in QEMU were in vain because libvirt now wants to start their own versioning system to give users more degrees of freedom even when you can't articulate a single concrete reason why users may want to do so. Regards, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Uh? I don't remember anybody suggesting changing CPU models on existing machines. We always tried to keep existing machines compatible. BTW your list does not answer my question. You would need examples where a CPU model changes between machines, and I am not aware of any example beyond the intentional -x.y variations. There are differences between KVM and TCG though, did you mean that? i440fx and q35 should be identical and isa-pc, too, and none anyway. None of this has anything to do with the host CPU. We are talking about the -x.y variations (that, yes, are intentional). But the fact that CPU features change (even the intentional ones in the -x.y machine variations) affect runnability of VMs (because enabling new CPU features in KVM require it to be supported by the host kernel code and by the host CPU). I was not thinking about the KVM and TCG differences, but this may also help libvirt deal with the KVM and TCG differences if necessary. I don't know what you mean by i440fx and q35 should be identical above. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. Regards, Daniel But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 05:56:35PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 04:51:00PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. Expanding on that, but tieing the CPU model to the machine type, QEMU has in turn effectively tied the machine type to the host hardware. eg, switching to a newer machine type, may then prevent the guest from being able to launch on the hardware that it was previously able to run on, due to some new requirement of the CPU model associated with the machine type. So why not keep machine type stable? There are many reasons to choose a particular machine type - for example, to achieve migration compat between hosts with different QEMU versions, or to enable access to some performance or bug fix in the machine type in question. Users / apps need to be free to make those decisions, without being restricted by changes in the CPU model which may affect what hardware the machine type can be used on. The current use of machine types for CPU model versioning is placing users between a rock hard place, giving them impossible decisions about which bad behaviour/bug they're willing to accept. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... BTW your list does not answer my question. You would need examples where a CPU model changes between machines, and I am not aware of any example beyond the intentional -x.y variations. There are differences between KVM and TCG though, did you mean that? i440fx and q35 should be identical and isa-pc, too, and none anyway. None of this has anything to do with the host CPU. Regards, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 06:40:19PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:25 schrieb Daniel P. Berrange: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. This coupling is by design and we expect all KVM/QEMU users to adhere to it, including those that use the libvirt tool (which I assume is going to be the majority of KVM users). Either you want a certain backwards-compatible machine and CPU, or you want the latest and greatest - why in the world mix and match?! As mentioned, changes/fixes to the CPU model can affect the ability to launch the guest on a particular host, so we need the ability to control when those CPU changes are activated for a guest, separately from the machine type. Would a qemu64-2.3 model help here that pc*-2.3 could use? I believe that's been proposed in the past. I don't oppose the idea of a fully-custom CPU, but this blatant attempt of ignoring QEMU's CPU versioning by libvirt worries me. That is still tieing CPU model names to machine type versions, unless I'm mis-understanding you. In general having QEMU models avialable vary depending on the QEMU version is creating problems for apps higher up the stack. By allowing libvirt to define the CPU model policy, it can also provide a more consistent interface to applications above, such as OpenStack, which will facilitate work in the schedular when it picks hosts capable of deploying/migrating a VM, when there are heterogenous QEMU versions across the hosts. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 18:53 schrieb Daniel P. Berrange: On Tue, Jun 23, 2015 at 06:40:19PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:25 schrieb Daniel P. Berrange: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. This coupling is by design and we expect all KVM/QEMU users to adhere to it, including those that use the libvirt tool (which I assume is going to be the majority of KVM users). Either you want a certain backwards-compatible machine and CPU, or you want the latest and greatest - why in the world mix and match?! As mentioned, changes/fixes to the CPU model can affect the ability to launch the guest on a particular host, so we need the ability to control when those CPU changes are activated for a guest, separately from the machine type. Why? Today's libvirt with QEMU 2.3 resolves pc machine to pc-i440fx-2.3 and the guest XML stays that way. When we add new features for 2.4, 2.3 is guaranteed to stay compatible. Any change would involve the libvirt user actively switching from pc-i440fx-2.3 to a different machine such as upcoming pc-i440fx-2.4. Why do you need to change the CPU separately? Why would a user want to run 2.2's CPU with a 2.3 machine? Or a 2.3 machine with a 2.4 CPU? That's nonsense. If you want to tweak features, you already have command line options available to do so on the basis of what the selected machine provides. Would a qemu64-2.3 model help here that pc*-2.3 could use? I believe that's been proposed in the past. I don't oppose the idea of a fully-custom CPU, but this blatant attempt of ignoring QEMU's CPU versioning by libvirt worries me. That is still tieing CPU model names to machine type versions, unless I'm mis-understanding you. In general having QEMU models avialable vary depending on the QEMU version is creating problems for apps higher up the stack. By allowing libvirt to define the CPU model policy, it can also provide a more consistent interface to applications above, such as OpenStack, which will facilitate work in the schedular when it picks hosts capable of deploying/migrating a VM, when there are heterogenous QEMU versions across the hosts. If you have heterogeneous QEMUs across hosts, then you need a common-denominator machine anyway because QEMU wouldn't start with a machine it doesn't know. Please give one concrete example of a real-world problem instead of hypothetical abstract combinations users may or may not be able to construct. Regards, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 07:18:06PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:08 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:44:57PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:38 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? I didn't take the time to count them, but I bet most of the commits I listed on my previous e-mail message are not bug fixes, but new features. Huh? Of course the latest machine model get new features. The point is that the previous ones don't and that's what we are providing them for - libvirt is expected to choose one machine and the contract with QEMU is that for that machine the CPU does *not* grow new features, and we're going at great lengths to achieve that. So this thread feels more and more weird... We are not talking about changes to existing machines. We are talking about having changes introduced in new machines (the one we did on purpose) affecting the runnability of the VM. You are talking abstract! Example 1: Point A: Machine pc-i440fx-2.3 exists Runs or runs not. Point B: Machine pc-i440fx-2.3 still exists Still runs or runs not due to guest ABI stability rules. Example 2: Point A: pc-i440fx-2.4 does not exist in 2.3 Does not run becomes it doesn't exist. Point B: New pc-i440fx-2.4 Runs or does not run, and if so has more features than pc-i440fx-2.3. There is no runnability problem - either it runs or it doesn't, but there's no change over time. This is what the machine -x.y versioning is all about. Consider a host currently running QEMU 2.3 with machine type pc-i440fx-2.3 used with SandyBridge. Now consider the def of SandyBridge was buggy and so in QEMU 2.4 we add the missing CPU feature flag 'wizz', and only enable that new feature flag with pc-i440fx-2.4 Now consider there was a bug in the virtio-scsi driver that we also fixed in QEMU 2.4 and thus pc-i440fx-2.4 includes that fix. Updating from pc-i440fx-2.3 to pc-i440fx-2.4 has a dependancy on the host CPU including the 'wizz' flag that was added. This new CPU feature can prevent the user from using the new machine type to get the virtio-scsi bug fix. Separately versioning CPU models still lets us preserve a stable guest ABI by default, but allows more flexibility when choosing to opt out of the ABI for a particular upgrade. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 07:55:14PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:42 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 07:29:15PM +0200, Andreas Färber wrote: In summary you seem to be saying that all the years we have spent fiddling around with those mind-boggling compat_props in QEMU were in vain because libvirt now wants to start their own versioning system to give users more degrees of freedom even when you can't articulate a single concrete reason why users may want to do so. I had a similar reaction when I learned about this libvirt expectation/requirement I was never aware of. But we spent lots of effort trying to do things differently doesn't seem like a valid justification for design decision. True, but my expectation is that libvirt is a friendly wrapper around QEMU using the mechanisms like -x.y we implemented for it and does not stand up at random and says, like Dan, that it wants to have things differently from now on, breaking all past assumptions and guarantees. FWIW, this isn't actually a change in opinion from the libvirt side. We never wanted CPU representation tied to machine types, even years back when the first qemu discussions in this area took place. We always wanted to be able to control CPU definition separately from the machine type. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 19:42 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 07:29:15PM +0200, Andreas Färber wrote: In summary you seem to be saying that all the years we have spent fiddling around with those mind-boggling compat_props in QEMU were in vain because libvirt now wants to start their own versioning system to give users more degrees of freedom even when you can't articulate a single concrete reason why users may want to do so. I had a similar reaction when I learned about this libvirt expectation/requirement I was never aware of. But we spent lots of effort trying to do things differently doesn't seem like a valid justification for design decision. True, but my expectation is that libvirt is a friendly wrapper around QEMU using the mechanisms like -x.y we implemented for it and does not stand up at random and says, like Dan, that it wants to have things differently from now on, breaking all past assumptions and guarantees. Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 07:41:50PM +0200, Andreas Färber wrote: Only if you use -cpu ...,enforce, no? The KVM feature filtering should take care of dropping features that are not available otherwise. So we seem to be getting to the interesting case of the same machine (different from what was said previously!) but different hosts. The QOM property gives you insights into which feature bits are set for the machine for the model (and for s390x I saw QMP extensions to the same effect, I thought). That way you could discover features to disable. However you'll only ever know which ones work once you've tried it once, right? On this subject, a big problem in QOM in general is that it hasn't included a distinction between object classes and object instances. So there's no way for us to introspect what a machine type provides without actually instantiating a guest with that machine type. Again allowing libvirt to control the CPU model removes this problem as libvirt will be able to determine what CPUs it can run on a given host without having to probe all the different CPU - machine type combinations for the QEMU on that host has. This in turn simplifies life for apps using libvirt, as they will always know that CPUFoo-X.Y will always mean the exact same thing on all the hosts they have no matter what QEMU version/machine is in use. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 21:25 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 08:35:54PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:39 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 07:18:06PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:08 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:44:57PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:38 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? I didn't take the time to count them, but I bet most of the commits I listed on my previous e-mail message are not bug fixes, but new features. Huh? Of course the latest machine model get new features. The point is that the previous ones don't and that's what we are providing them for - libvirt is expected to choose one machine and the contract with QEMU is that for that machine the CPU does *not* grow new features, and we're going at great lengths to achieve that. So this thread feels more and more weird... We are not talking about changes to existing machines. We are talking about having changes introduced in new machines (the one we did on purpose) affecting the runnability of the VM. You are talking abstract! I am just talking about a different problem, and I don't know if you are purposely trying to ignore it, or are just denying that it is a problem. So, are you and Dan talking about the same problem or different ones? The same one. I am not deliberately ignoring anything here, but I am denying there is a problem until either of you explains what a concrete problem is. Seems we are slowly getting there now. I hope so. :) Example 1: Point A: Machine pc-i440fx-2.3 exists Runs or runs not. Point B: Machine pc-i440fx-2.3 still exists Still runs or runs not due to guest ABI stability rules. If you didn't change the machine name, this is not the problem we are talking about. OK. Example 2: Point A: pc-i440fx-2.4 does not exist in 2.3 Does not run becomes it doesn't exist. Point B: New pc-i440fx-2.4 Runs or does not run, and if so has more features than pc-i440fx-2.3. If you didn't change the machine name, this is not the problem we are talking about. There is no runnability problem - either it runs or it doesn't, but there's no change over time. This is what the machine -x.y versioning is all about. Let's try a concrete example: * User is running a kernel that can't emulate x2apic * User is running pc-i440fx-1.7 * User wants the gigabyte alignment change implemented by commit bb43d3839c29b17a2f5c122114cd4ca978065a18 * User changes machine to pc-i440fx-2.0 * x2apic is now enabled by default in all CPU models * VM with the same configuration (just the machine change) is not runnable anymore in the same host Then let's take a step back: In order to change the machine type, the user shuts the machine down (it does not run!), edits the XML and tries to boot it up again. That's where I've challenged your use of the term of changed runnability above. I acknowledged, it might happen that it does not run. But that has nothing to do with compatibility of QEMU versions v2.3.0 vs. v2.4.0 then, it is the user's active choice of options that are incompatible with her system and that never before worked there. That seems perfectly valid and unavoidable, just like adding a non-existing command-line option or an unknown XML element to the guest config. If you add a non-existing command-line option or unknown XML element, you are providing bad input. The most obvious way to handle it is an error. If you change -machine, you are providing good input, but the user won't have a good explanation why it can't run (because it is a perfectly valid machine name, reported as supported by QEMU). Or maybe they will see an explanation, but will have no idea what they need to change to make the new machine runnable. CPU model runnability, on the other hand, is well documented in the libvirt API, and would even allow a management system to automatically find a solution (because it can tell exactly
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 08:35:54PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:39 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 07:18:06PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:08 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:44:57PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:38 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? I didn't take the time to count them, but I bet most of the commits I listed on my previous e-mail message are not bug fixes, but new features. Huh? Of course the latest machine model get new features. The point is that the previous ones don't and that's what we are providing them for - libvirt is expected to choose one machine and the contract with QEMU is that for that machine the CPU does *not* grow new features, and we're going at great lengths to achieve that. So this thread feels more and more weird... We are not talking about changes to existing machines. We are talking about having changes introduced in new machines (the one we did on purpose) affecting the runnability of the VM. You are talking abstract! I am just talking about a different problem, and I don't know if you are purposely trying to ignore it, or are just denying that it is a problem. So, are you and Dan talking about the same problem or different ones? The same one. I am not deliberately ignoring anything here, but I am denying there is a problem until either of you explains what a concrete problem is. Seems we are slowly getting there now. I hope so. :) Example 1: Point A: Machine pc-i440fx-2.3 exists Runs or runs not. Point B: Machine pc-i440fx-2.3 still exists Still runs or runs not due to guest ABI stability rules. If you didn't change the machine name, this is not the problem we are talking about. OK. Example 2: Point A: pc-i440fx-2.4 does not exist in 2.3 Does not run becomes it doesn't exist. Point B: New pc-i440fx-2.4 Runs or does not run, and if so has more features than pc-i440fx-2.3. If you didn't change the machine name, this is not the problem we are talking about. There is no runnability problem - either it runs or it doesn't, but there's no change over time. This is what the machine -x.y versioning is all about. Let's try a concrete example: * User is running a kernel that can't emulate x2apic * User is running pc-i440fx-1.7 * User wants the gigabyte alignment change implemented by commit bb43d3839c29b17a2f5c122114cd4ca978065a18 * User changes machine to pc-i440fx-2.0 * x2apic is now enabled by default in all CPU models * VM with the same configuration (just the machine change) is not runnable anymore in the same host Then let's take a step back: In order to change the machine type, the user shuts the machine down (it does not run!), edits the XML and tries to boot it up again. That's where I've challenged your use of the term of changed runnability above. I acknowledged, it might happen that it does not run. But that has nothing to do with compatibility of QEMU versions v2.3.0 vs. v2.4.0 then, it is the user's active choice of options that are incompatible with her system and that never before worked there. That seems perfectly valid and unavoidable, just like adding a non-existing command-line option or an unknown XML element to the guest config. If you add a non-existing command-line option or unknown XML element, you are providing bad input. The most obvious way to handle it is an error. If you change -machine, you are providing good input, but the user won't have a good explanation why it can't run (because it is a perfectly valid machine name, reported as supported by QEMU). Or maybe they will see an explanation, but will have no idea what they need to change to make the new machine runnable. CPU model runnability, on the other hand, is well documented in the libvirt API, and would even allow a management system to automatically find a solution
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 19:45 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 07:41:50PM +0200, Andreas Färber wrote: [...] If that is the whole problem here, then why not just add a global flag to only enable explicitly requested KVM features? All other features should not depend on the host, and the whole discussion about -x.y seems like a distraction. Now replace KVM features with CPU fatures, because all CPU features are KVM features, as all of them depend on KVM code enabling them on GET_SUPPORTED_CPUID. Thus, the global flag to only enable explicitly request KVM features on CPUs is -cpu custom, which doesn't enable any CPU feature at all. If libvirt wants to use an empty CPU model, then why export our models to libvirt? I don't mind there being an optional custom model, I mind our compat_props getting ignored that way, which are unrelated to adding new features, in fact they suppress just that for the -2.3 examples. Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton; HRB 21284 (AG Nürnberg)
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 07:58:44PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:45 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 07:41:50PM +0200, Andreas Färber wrote: [...] If that is the whole problem here, then why not just add a global flag to only enable explicitly requested KVM features? All other features should not depend on the host, and the whole discussion about -x.y seems like a distraction. Now replace KVM features with CPU fatures, because all CPU features are KVM features, as all of them depend on KVM code enabling them on GET_SUPPORTED_CPUID. Thus, the global flag to only enable explicitly request KVM features on CPUs is -cpu custom, which doesn't enable any CPU feature at all. If libvirt wants to use an empty CPU model, then why export our models to libvirt? I don't mind there being an optional custom model, I mind our compat_props getting ignored that way, which are unrelated to adding new features, in fact they suppress just that for the -2.3 examples. If QEMU has a '-cpu custom' there isn't actually any need for QEMU to export its CPU models to libvirt. Libvirt already has its own CPU model database, so we'd just start to use our own database exclusively. Libvirt would have to take care that we don't break ABI when updating from a QEMU that pre-dates the '-cpu custom' feature - for that we have done a one-time only extraction of details of all the historical variations in QEMU CPUS models per machine type. So apart from backcompat for existing QEMUs in the wild, libvirt would no longer have any need to know about QEMU's CPU models. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 07:58:44PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:45 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 07:41:50PM +0200, Andreas Färber wrote: [...] If that is the whole problem here, then why not just add a global flag to only enable explicitly requested KVM features? All other features should not depend on the host, and the whole discussion about -x.y seems like a distraction. Now replace KVM features with CPU fatures, because all CPU features are KVM features, as all of them depend on KVM code enabling them on GET_SUPPORTED_CPUID. Thus, the global flag to only enable explicitly request KVM features on CPUs is -cpu custom, which doesn't enable any CPU feature at all. If libvirt wants to use an empty CPU model, then why export our models to libvirt? We wouldn't need it anymore. We would probably keep it just because old libvirt versions (or maybe the non-x86 libvirt code) may use it, or maybe libvirt will keep using the existing QEMU models for existing VMs (because that would be simpler than converting existing -cpu Foo to -cpu custom -readconfig). I don't mind there being an optional custom model, I mind our compat_props getting ignored that way, which are unrelated to adding new features, in fact they suppress just that for the -2.3 examples. I am also not happy for having spent lots of time on compat_props for CPUs when it is not going to be needed by libvirt anymore, but that's a sunk cost. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Am 23.06.2015 um 19:39 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 07:18:06PM +0200, Andreas Färber wrote: Am 23.06.2015 um 19:08 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:44:57PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:38 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? I didn't take the time to count them, but I bet most of the commits I listed on my previous e-mail message are not bug fixes, but new features. Huh? Of course the latest machine model get new features. The point is that the previous ones don't and that's what we are providing them for - libvirt is expected to choose one machine and the contract with QEMU is that for that machine the CPU does *not* grow new features, and we're going at great lengths to achieve that. So this thread feels more and more weird... We are not talking about changes to existing machines. We are talking about having changes introduced in new machines (the one we did on purpose) affecting the runnability of the VM. You are talking abstract! I am just talking about a different problem, and I don't know if you are purposely trying to ignore it, or are just denying that it is a problem. So, are you and Dan talking about the same problem or different ones? I am not deliberately ignoring anything here, but I am denying there is a problem until either of you explains what a concrete problem is. Seems we are slowly getting there now. Example 1: Point A: Machine pc-i440fx-2.3 exists Runs or runs not. Point B: Machine pc-i440fx-2.3 still exists Still runs or runs not due to guest ABI stability rules. If you didn't change the machine name, this is not the problem we are talking about. OK. Example 2: Point A: pc-i440fx-2.4 does not exist in 2.3 Does not run becomes it doesn't exist. Point B: New pc-i440fx-2.4 Runs or does not run, and if so has more features than pc-i440fx-2.3. If you didn't change the machine name, this is not the problem we are talking about. There is no runnability problem - either it runs or it doesn't, but there's no change over time. This is what the machine -x.y versioning is all about. Let's try a concrete example: * User is running a kernel that can't emulate x2apic * User is running pc-i440fx-1.7 * User wants the gigabyte alignment change implemented by commit bb43d3839c29b17a2f5c122114cd4ca978065a18 * User changes machine to pc-i440fx-2.0 * x2apic is now enabled by default in all CPU models * VM with the same configuration (just the machine change) is not runnable anymore in the same host Then let's take a step back: In order to change the machine type, the user shuts the machine down (it does not run!), edits the XML and tries to boot it up again. That's where I've challenged your use of the term of changed runnability above. I acknowledged, it might happen that it does not run. But that has nothing to do with compatibility of QEMU versions v2.3.0 vs. v2.4.0 then, it is the user's active choice of options that are incompatible with her system and that never before worked there. That seems perfectly valid and unavoidable, just like adding a non-existing command-line option or an unknown XML element to the guest config. The difference of opinion seems to be that when there is a bug in QEMU, I require that the user updates QEMU (not necessarily to a new version), whereas you are proposing that libvirt should be the one to work around bugs in QEMU by tweaking command line parameters. In order to get a virtio-scsi or gigabyte alignment fix that varies across -x.y machines, that feature can just as well be enabled via global properties on the old machine. New machines are primarily for new features. If someone wants to use that new -2.0 machine, they need to pass the correct options such as ,-x2apic in your example or use a CPU model that does not enable such options by default. (FWIW in that concrete example I remember Paolo(?) saying that that feature had been supported for a really long time already.) The user, who actively
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 09:41:51PM +0200, Andreas Färber wrote: [...] I am going to stop arguing here and suggest you put this on the agenda for the next KVM call. I am a bit confused. You said I don't mind there being an optional custom model in a previous message. If you have objections to libvirt API design decisions, I understand it, but I suggest you take them to the libvir-list mailing list. Now, if you have objections to having an optional custom model (i.e. valid reasons to not apply patch 1/2), please let me know. I didn't see a single argument to reject the patch, yet. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 06:13:24PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:47:16PM +0200, Andreas Färber wrote: Am 23.06.2015 um 18:42 schrieb Daniel P. Berrange: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? Huh, I never said we wanted to bring back bugs. This is about allowing libvirt to fix the CPU bugs in a way that is independant of the machine types and portable across hypervisors we deal with. We're absolutely still going to fix CPU model bugs and ensure stable guest ABI. No, that's contradictory! Through the -x.y machines we leave bugs in the old models *exactly* to assure a stable guest ABI. Fixes are only be applied to new machines, thus I'm pointing out that you should not use a new CPU model with an old machine type. I'm not saying that libvirt would ever allow a silent guest ABI change. Given a libvirt XML config, the guest ABI will never change without an explicit action on the part of the app/user to change the XML. This is all about dealing with the case where the app / user conciously needs/wants to opt-in to a guest ABI change for the guest. eg they wish to make use of some bug fix or feature improvement in the new machine type, but they do *not* wish to have the CPU model changed, because of some CPU model change that is incompatible with their hosts' CPUs. Conversely, they may wish to get access to a new CPU model, but not wish to have the rest of the guest ABI change. In both cases the user is explicitly opt-ing in the ABI change with knowledge about what this might mean for the guest OS. Currently we are tieing users hands by forcing CPU and machine types to change in lockstep. Regards, Daniel Can we have a specific example please? It's hard to understand the facts based on such generic statements. -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 05:26:38PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 09:41:51PM +0200, Andreas Färber wrote: [...] Given that we have had this versioning system for years and no problem specifically with 2.4 has been raised, I see this as 2.5+ material at this point. I see this on 2.4 schedule: 2015-06-16 Soft feature freeze. All features should have patches on the list by this date; major features should have initial code committed. It is a 9-line patch (from which 6 are declarations, and 2 are actual code statements), it doesn't affect anybody who is not explicitly using -cpu custom, it is an useful feature, and it has been on the list since June 8 (and as RFC since April 13). Unless somebody gives me a good reason to consider it harmful (that's different from not accepting the explanations why it is useful, or disagreeing with users that want to use it), I would like to include it in 2.4 as long as I get Reviewed-by lines before hard freeze. For the record I have nothing against the patch itself. Might be useful for testing or something. And how libvirt uses QEMU is, in the end, in the hands of libvirt developers. I do care about command line stability generally, and about people not breaking existing documentation and tools using QEMU directly as opposed to through libvirt, that's the only reason I participated in this discussion. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 05:42:04PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:33:05PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 05:25:55PM +0100, Daniel P. Berrange wrote: On Tue, Jun 23, 2015 at 06:15:51PM +0200, Andreas Färber wrote: Am 23.06.2015 um 17:58 schrieb Eduardo Habkost: On Tue, Jun 23, 2015 at 05:32:42PM +0200, Michael S. Tsirkin wrote: On Tue, Jun 23, 2015 at 12:08:28PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 02:32:00PM +0200, Andreas Färber wrote: Am 08.06.2015 um 22:18 schrieb Jiri Denemark: To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. That sounds like FUD to me. Any concrete data points where QEMU does not have a stable ABI for x86 CPUs? That's what we have the pc*-x.y machines for. What Jiri is saying that the CPUs change depending on -mmachine, not that the ABI is broken by a given machine. The problem here is that libvirt needs to provide CPU models whose runnability does not depend on the machine-type. If users have a VM that is running in a host and the VM machine-type changes, How does it change, and why? Sometimes we add features to a CPU model because they were not emulated by KVM and now they are. Sometimes we remove or add features or change other fields because we are fixing previous mistakes. Recently we we were going to remove features from models because of an Intel CPU errata, but then decided to create a new CPU model name instead. See some examples at the end of this message. the VM should be still runnable in that host. QEMU doesn't provide that, our CPU models may change when we introduce new machine-types, so we are giving them a mechanism that allows libvirt to implement the policy they need. I don't mind wrt CPU specifically, but we absolutely do change guest ABI in many ways when we change machine types. All the other ABI changes we introduce in QEMU don't affect runnability of the VM in a given host, that's the problem we are trying to address here. ABI changes are expected when changing to a new machine, runnability changes aren't. Examples of commits changing CPU models: [snip] I've always advocated remaining backwards-compatible and only making CPU model changes for new machines. You among others felt that was not always necessary, and now you're using the lack thereof as an argument to stop using QEMU's CPU models at all? That sounds convoluted... Whether QEMU changed the CPU for existing machines, or only for new machines is actually not the core problem. Even if we only changed the CPU in new machines that would still be an unsatisfactory situation because we want to be able to be able to access different versions of the CPU without the machine type changing, and access different versions of the machine type, without the CPU changing. IOW it is the fact that the changes in CPU are tied to changes in machine type that is the core problem. But that's because we are fixing bugs. If CPU X used to work on hardware Y in machine type A and stopped in machine type B, this is because we have determined that it's the right thing to do for the guests and the users. We don't break stuff just for fun. Why do you want to bring back the bugs we fixed? Huh, I never said we wanted to bring back bugs. This is about allowing libvirt to fix the CPU bugs in a way that is independant of the machine types and portable across hypervisors we deal with. We're absolutely still going to fix CPU model bugs and ensure stable guest ABI. Regards, Daniel So any single CPU flag now needs to be added in - kvm - qemu - libvirt Next thing libvirt will decide it's a policy thing and so needs to be pushed up to openstack. We should just figure out what you want to do and support it in QEMU. Are there many examples where a single flag used to work and then stopped? I would say such a change is problematic anyway: not everyone uses libvirt, you are breaking things for people that run
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 02:11:22PM -0300, Eduardo Habkost wrote: Even if it is a bug fix. If it is a change that can make the VM unrunnable, it needs to be controlled by a separate flag, not by the machine-type. I agree - command line compatibility is important. But we are supposed to provide that. I am surprised that libvirt suddenly wants to avoid some command line flags because they are not stable. IMHO we did something wrong here if so. Maybe there was a valid reason for it. But then won't it apply to libvirt as well? Now, if people want to update CPU models outside the QEMU binary, that might be doable simply by moving them to a separate package, with a text file that QEMU reads at startup. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 09:41:51PM +0200, Andreas Färber wrote: [...] Given that we have had this versioning system for years and no problem specifically with 2.4 has been raised, I see this as 2.5+ material at this point. I see this on 2.4 schedule: 2015-06-16 Soft feature freeze. All features should have patches on the list by this date; major features should have initial code committed. It is a 9-line patch (from which 6 are declarations, and 2 are actual code statements), it doesn't affect anybody who is not explicitly using -cpu custom, it is an useful feature, and it has been on the list since June 8 (and as RFC since April 13). Unless somebody gives me a good reason to consider it harmful (that's different from not accepting the explanations why it is useful, or disagreeing with users that want to use it), I would like to include it in 2.4 as long as I get Reviewed-by lines before hard freeze. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 23, 2015 at 02:42:37PM -0300, Eduardo Habkost wrote: On Tue, Jun 23, 2015 at 07:29:15PM +0200, Andreas Färber wrote: In summary you seem to be saying that all the years we have spent fiddling around with those mind-boggling compat_props in QEMU were in vain because libvirt now wants to start their own versioning system to give users more degrees of freedom even when you can't articulate a single concrete reason why users may want to do so. I had a similar reaction when I learned about this libvirt expectation/requirement I was never aware of. But we spent lots of effort trying to do things differently doesn't seem like a valid justification for design decision. Users will be hurt because they'll run untested configurations seems like a valid reason. -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
Ping? Any feedback? I want to get this into 2.4. On Mon, Jun 08, 2015 at 04:07:38PM -0300, Eduardo Habkost wrote: The problem: The existing libvirt APIs assume that if a given CPU model is runnable in a host kernel+hardware combination, it will be always runnable on that host even if the machine-type changes. That assumption is implied in some of libvirt interfaces, for example, at: 1) Host capabilities, which let callers know the set of CPU models that can run in a host: https://libvirt.org/formatcaps.html#elementHost virsh capabilities returns a CPU model name + CPU feature list, assuming that a CPU model name has a meaning that's independent from the machine-type. 2) The function that checks if a given CPU model definition is compatible with a host (virConnectCompareCPU()), which does not take the machine-type as argument: http://libvirt.org/html/libvirt-libvirt-host.html#virConnectCompareCPU But that assumption is not true, as QEMU changes CPU models in new machine-types when fixing bugs, or when new features (previously unsupported by QEMU, TCG or KVM) get implemented. The solution: libvirt can solve this problem partially by making sure every feature in a CPU model is explicitly configured, instead of (incorrectly) expecting that a named CPU model will never change in QEMU. But this doesn't solve the problem completely, because it is still possible that new features unknown to libvirt get enabled in the default CPU model in future machine-types (that's very likely to happen when we introduce new KVM features, for example). So, to make sure no new feature will be ever enabled without the knowledge of libvirt, add a -cpu custom mode, where no CPU model data is loaded at all, and everything needs to be configured explicitly using CPU properties. That means no CPU features will ever change depending on machine-type or accelerator capabilities when using -cpu custom. * * * I know that this is basically the opposite of what we were aiming at in the last few month^Wyears, where we were struggling to implement probing interfaces to expose machine-type-dependent data for libvirt. But, at least the fact that we could implement the new approach using a 9-line patch means we were still going in the right direction. :) To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. * * * This is basically the same version I sent as an RFC in April. A git tree is available at: git://github.com/ehabkost/qemu-hacks.git work/x86-cpu-custom-model Eduardo Habkost (2): target-i386: Introduce -cpu custom scripts: x86-cpu-model-dump script scripts/x86-cpu-model-dump | 322 + target-i386/cpu.c | 10 +- 2 files changed, 331 insertions(+), 1 deletion(-) create mode 100755 scripts/x86-cpu-model-dump -- 2.1.0 -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Mon, Jun 08, 2015 at 10:18:35PM +0200, Jiri Denemark wrote: On Mon, Jun 08, 2015 at 16:07:38 -0300, Eduardo Habkost wrote: ... libvirt can solve this problem partially by making sure every feature in a CPU model is explicitly configured, instead of (incorrectly) expecting that a named CPU model will never change in QEMU. But this doesn't solve the problem completely, because it is still possible that new features unknown to libvirt get enabled in the default CPU model in future machine-types (that's very likely to happen when we introduce new KVM features, for example). So, to make sure no new feature will be ever enabled without the knowledge of libvirt, add a -cpu custom mode, where no CPU model data is loaded at all, and everything needs to be configured explicitly using CPU properties. That means no CPU features will ever change depending on machine-type or accelerator capabilities when using -cpu custom. * * * I know that this is basically the opposite of what we were aiming at in the last few month^Wyears, where we were struggling to implement probing interfaces to expose machine-type-dependent data for libvirt. But, at least the fact that we could implement the new approach using a 9-line patch means we were still going in the right direction. :) To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. I meanwhile, always wanted the full CPU config data in libvirt, so that we could ensure libvirt was able to use the exact same CPU model setup on other hypervisors too - eg Xen, VMWare let us specify the CPUID masks so we could re-use the libvirt data there. I will summarize my ideas on how libvirt should be using -cpu custom and send them to libvirt-list soon. These patches are x86 obviously - is there anything we need to worry about for non-x86 architectures I wonder ? IIRC, all the non-x86 archs have traditionally only needed CPU model names and don't really have the same level of per-flag CPUID mask configurability that x86 has. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Tue, Jun 09, 2015 at 09:56:25AM +0100, Daniel P. Berrange wrote: On Mon, Jun 08, 2015 at 10:18:35PM +0200, Jiri Denemark wrote: On Mon, Jun 08, 2015 at 16:07:38 -0300, Eduardo Habkost wrote: ... libvirt can solve this problem partially by making sure every feature in a CPU model is explicitly configured, instead of (incorrectly) expecting that a named CPU model will never change in QEMU. But this doesn't solve the problem completely, because it is still possible that new features unknown to libvirt get enabled in the default CPU model in future machine-types (that's very likely to happen when we introduce new KVM features, for example). So, to make sure no new feature will be ever enabled without the knowledge of libvirt, add a -cpu custom mode, where no CPU model data is loaded at all, and everything needs to be configured explicitly using CPU properties. That means no CPU features will ever change depending on machine-type or accelerator capabilities when using -cpu custom. * * * I know that this is basically the opposite of what we were aiming at in the last few month^Wyears, where we were struggling to implement probing interfaces to expose machine-type-dependent data for libvirt. But, at least the fact that we could implement the new approach using a 9-line patch means we were still going in the right direction. :) To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. I meanwhile, always wanted the full CPU config data in libvirt, so that we could ensure libvirt was able to use the exact same CPU model setup on other hypervisors too - eg Xen, VMWare let us specify the CPUID masks so we could re-use the libvirt data there. I will summarize my ideas on how libvirt should be using -cpu custom and send them to libvirt-list soon. These patches are x86 obviously - is there anything we need to worry about for non-x86 architectures I wonder ? IIRC, all the non-x86 archs have traditionally only needed CPU model names and don't really have the same level of per-flag CPUID mask configurability that x86 has. X86 started with opaque CPU model names hiding implementation details, then moved to allow extra -cpu parameters. Then we noticed that the CPU models were hiding too much from libvirt in the X86 case, and now those parameters were converted to become QOM properties configurable using -global. I expect other architectures to follow a similar pattern and allow things to be configured using QOM properties, but I am not sure they would go to the extreme of making every single detail configurable using QOM. One thing you may need to worry about for all architectures is to check if a CPU model is runnable in a host before starting or migrating a VM. In this case, we're introducing a generic mechanism in query-cpu-definitions for that. See [PATCH v6 15/17] target-s390x: Extend arch specific QMP command query-cpu-definitions and related patches (posted on 2015-04-27) on qemu-devel. And in the case of runnability checks, x86 is a bit more complex, too (because it is too configurable in QEMU and in libvirt): as libvirt needs to know what exactly is blocking the CPU from running, we have the filtered-features property (that libvirt can start using, now that we have the qom-path field on query-cpus). -- Eduardo
Re: [Qemu-devel] [PATCH 0/2] target-i386: custom CPU model + script to dump existing CPU models
On Mon, Jun 08, 2015 at 16:07:38 -0300, Eduardo Habkost wrote: ... libvirt can solve this problem partially by making sure every feature in a CPU model is explicitly configured, instead of (incorrectly) expecting that a named CPU model will never change in QEMU. But this doesn't solve the problem completely, because it is still possible that new features unknown to libvirt get enabled in the default CPU model in future machine-types (that's very likely to happen when we introduce new KVM features, for example). So, to make sure no new feature will be ever enabled without the knowledge of libvirt, add a -cpu custom mode, where no CPU model data is loaded at all, and everything needs to be configured explicitly using CPU properties. That means no CPU features will ever change depending on machine-type or accelerator capabilities when using -cpu custom. * * * I know that this is basically the opposite of what we were aiming at in the last few month^Wyears, where we were struggling to implement probing interfaces to expose machine-type-dependent data for libvirt. But, at least the fact that we could implement the new approach using a 9-line patch means we were still going in the right direction. :) To help libvirt in the transition, a x86-cpu-model-dump script is provided, that will generate a config file that can be loaded using -readconfig, based on the -cpu and -machine options provided in the command-line. Thanks Eduardo, I never was a big fan of moving (or copying) all the CPU configuration data to libvirt, but now I think it actually makes sense. We already have a partial copy of CPU model definitions in libvirt anyway, but as QEMU changes some CPU models in some machine types (and libvirt does not do that) we have no real control over the guest CPU configuration. While what we really want is full control to enforce stable guest ABI. I will summarize my ideas on how libvirt should be using -cpu custom and send them to libvirt-list soon. Jirka