Re: [libvirt] [RFC] Support for CPUID masking
On 10/13/2009 11:40 AM, Daniel P. Berrange wrote: On Tue, Oct 13, 2009 at 02:56:41AM -0400, john cooper wrote: Dor Laor wrote: What about another approach for the cpuid issue: I think that dealing with specific flags is pretty error prone on all levels - virt-mgr, libvirt, qemu, migration, and even the guest. ..and performance verification, QA, and the average end user. Unless we reduce all possible combinations of knob settings into a few well thought out lumped models the complexity can be overwhelming. That is a policy decision for applications to make. libvirt should expose the fine grained named CPU models + arbitrary flags, and other bits of info as appropriate (eg formal model for core/socket topology). Apps can decide whether they want to turn that into a higher level concept where admins pick one of a handful of common setups, or expose the full level of control As long as the cpu model is exposed (both host and guest) it works for me. My guess is that most apps will try to be as dumb as possible. Some might only use problematic flags like the NX bit that might be off in the bios menu. Daniel -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking
Dor Laor wrote: What about another approach for the cpuid issue: I think that dealing with specific flags is pretty error prone on all levels - virt-mgr, libvirt, qemu, migration, and even the guest. ..and performance verification, QA, and the average end user. Unless we reduce all possible combinations of knob settings into a few well thought out lumped models the complexity can be overwhelming. It is probably a reasonable compromise to initially support the most obvious use cases with more fringe models added on an as-needed, as-justified basis. We can't just 'invent' new cpu modules that will be a combination of flags we have on the host. It might work for some kernels/apps but it might break others. In addition to cpuid we have the stepping and more MSRs that needed to be hidden/exposed. In whatever abstraction, exposing all of that low-level detail as the sole configuration means will needlessly exaggerate the complexity as above. The offer: Libvirt, virt-mgr, qemu will not deal with lowlevel bits of the cpuid. We'll use predefined cpu modules to be emulated. These predefined modules will represent a real module that was once shipped by Intel/AMD. We'll write an additional tool, not under libvirt, that will be able to calculate all the virtual cpus that can run over the physical cpu. This tool will be the one messing with /proc/cpuinfo, etc. Example (theoretical): Physical cpu is Intel(R) Core(TM)2 Duo CPU T9600 @ 2.80GHz the output of the tool will be: shellcomputeVirtCpuCapabilities core2duo core solo 486 pentium3 .. Libvirt will only expose the real physical cpu and all of the outputs above. Higher level mgmt will compute the best -cpu to pick for the VM, and it will take account the user needs for performance or for migration flexibility. The cons: - Simple for all levels - Fast implementation - Accurate representative of real cpus The pros: - none - we should write the tool anyway - Maybe we hide some of the possibilities, but as said above, it is safer and friendlier. I think paving over the complexity to export the most common use cases is a reasonable approach. We can allow some sort of fingers-in-the-gears mode for experimentation and tuning as needed. But supporting features such as safe migration could be a non-goal in these scenarios. -john I also recommend of adding a field to be added next to the cpu for flexibility and possible future changes + dealing with problematic bios with NX bit disabled. Regards, Dor Thanks Mukesh On Thu, Sep 3, 2009 at 3:09 PM, Daniel P. Berrangeberra...@redhat.com wrote: On Thu, Sep 03, 2009 at 11:19:47AM +0300, Dor Laor wrote: On 09/02/2009 07:09 PM, Daniel P. Berrange wrote: On Wed, Sep 02, 2009 at 11:59:39AM -0400, Jim Paris wrote: Jiri Denemark wrote: Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. My proposal is to define new 'cpuid' feature advertised in guest capabilities: ... domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask ... What are your opinions about this? I think it's too low-level, and the structure is x86-specific. QEMU and KVM compute their CPUID response based on arguments to the -cpu argument, e.g.: -cpu core2duo,model=23,+ssse3,+lahf_lm I think a similar structure makes more sense for libvirt, where the configuration generally avoids big blocks of binary data, and the XML format should suit other architectures as well. I'm going backforth on this too. We essentially have 3 options - Named CPU + flags/features - CPUID masks - Allow either If we do either of the first two, we have to translate between the two formats for one or more of the hypervisors. For the last one we are just punting the problem off to applications. If we choose CPUID, and made QEMU driver convert to named CPU + flags we'd be stuck for non-x86 as you say. Why is that? cpu model + flags may apply for other arch too. If we have CPUID in the XML, there is no meaningful CPUID register representation for sparc/ppc/arm/etc. It is an x86 concept, which is almost certainly why QEMU uses named CPU models + named flags instead of CPUID as is public facing config. Xen/VMWare of course don't have this limitation since they only really care about x86. So really QEMU's CPU model + flags approach is more generic albeit being much more verbose to achieve the same level of expressivity. If we chose named CPU + flags, and made VMWare/Xen convert to raw CPUID we'd potentially loose information if user had defined a config with a raw CPUID mask outside context of libvirt. The other thing to remember is that CPUID also encodes sockets/cores/ threads
Re: [libvirt] [RFC] Support for CPUID masking
On Tue, Oct 13, 2009 at 02:56:41AM -0400, john cooper wrote: Dor Laor wrote: What about another approach for the cpuid issue: I think that dealing with specific flags is pretty error prone on all levels - virt-mgr, libvirt, qemu, migration, and even the guest. ..and performance verification, QA, and the average end user. Unless we reduce all possible combinations of knob settings into a few well thought out lumped models the complexity can be overwhelming. That is a policy decision for applications to make. libvirt should expose the fine grained named CPU models + arbitrary flags, and other bits of info as appropriate (eg formal model for core/socket topology). Apps can decide whether they want to turn that into a higher level concept where admins pick one of a handful of common setups, or expose the full level of control Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking
On Sun, Oct 11, 2009 at 11:44:04AM +0200, Dor Laor wrote: On 10/09/2009 12:58 PM, Mukesh G wrote: Speaking from an x86 angle,providing an ability to enable or disable high level constructs like SSE instead of low level constructs, will make it easy to understand. ain't SSE a low level construct too? What about another approach for the cpuid issue: I think that dealing with specific flags is pretty error prone on all levels - virt-mgr, libvirt, qemu, migration, and even the guest. We can't just 'invent' new cpu modules that will be a combination of flags we have on the host. It might work for some kernels/apps but it might break others. In addition to cpuid we have the stepping and more MSRs that needed to be hidden/exposed. The offer: Libvirt, virt-mgr, qemu will not deal with lowlevel bits of the cpuid. We'll use predefined cpu modules to be emulated. These predefined modules will represent a real module that was once shipped by Intel/AMD. We'll write an additional tool, not under libvirt, that will be able to calculate all the virtual cpus that can run over the physical cpu. This tool will be the one messing with /proc/cpuinfo, etc. Example (theoretical): Physical cpu is Intel(R) Core(TM)2 Duo CPU T9600 @ 2.80GHz the output of the tool will be: shellcomputeVirtCpuCapabilities core2duo core solo 486 pentium3 .. Libvirt will only expose the real physical cpu and all of the outputs above. Higher level mgmt will compute the best -cpu to pick for the VM, and it will take account the user needs for performance or for migration flexibility. The cons: - Simple for all levels - Fast implementation - Accurate representative of real cpus The pros: - none - we should write the tool anyway - Maybe we hide some of the possibilities, but as said above, it is safer and friendlier. I also recommend of adding a field to be added next to the cpu for flexibility and possible future changes + dealing with problematic bios with NX bit disabled. Regards, Dor Dan was discussing something quite like this early on, IIRC... --Hugh Thanks Mukesh On Thu, Sep 3, 2009 at 3:09 PM, Daniel P. Berrangeberra...@redhat.com wrote: On Thu, Sep 03, 2009 at 11:19:47AM +0300, Dor Laor wrote: On 09/02/2009 07:09 PM, Daniel P. Berrange wrote: On Wed, Sep 02, 2009 at 11:59:39AM -0400, Jim Paris wrote: Jiri Denemark wrote: Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. My proposal is to define new 'cpuid' feature advertised in guest capabilities: ... domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask ... What are your opinions about this? I think it's too low-level, and the structure is x86-specific. QEMU and KVM compute their CPUID response based on arguments to the -cpu argument, e.g.: -cpu core2duo,model=23,+ssse3,+lahf_lm I think a similar structure makes more sense for libvirt, where the configuration generally avoids big blocks of binary data, and the XML format should suit other architectures as well. I'm going backforth on this too. We essentially have 3 options - Named CPU + flags/features - CPUID masks - Allow either If we do either of the first two, we have to translate between the two formats for one or more of the hypervisors. For the last one we are just punting the problem off to applications. If we choose CPUID, and made QEMU driver convert to named CPU + flags we'd be stuck for non-x86 as you say. Why is that? cpu model + flags may apply for other arch too. If we have CPUID in the XML, there is no meaningful CPUID register representation for sparc/ppc/arm/etc. It is an x86 concept, which is almost certainly why QEMU uses named CPU models + named flags instead of CPUID as is public facing config. Xen/VMWare of course don't have this limitation since they only really care about x86. So really QEMU's CPU model + flags approach is more generic albeit being much more verbose to achieve the same level of expressivity. If we chose named CPU + flags, and made VMWare/Xen convert to raw CPUID we'd potentially loose information if user had defined a config with a raw CPUID mask outside context of libvirt. The other thing to remember is that CPUID also encodes sockets/cores/ threads topology data, and it'd be very desirable to expose that in a sensible fashion (ie not a bitmask). On balance i'm currently leaning to named CPU + flags + expliciti topology data because although its harder to implement for Xen/VMWare I think its much nicer to applicationsusers. We might loose a tiny bit of data in the CPU -named/flags conversion for Xen/VMWare but I reckon we can get it good enough
Re: [libvirt] [RFC] Support for CPUID masking
On 10/09/2009 12:58 PM, Mukesh G wrote: Speaking from an x86 angle,providing an ability to enable or disable high level constructs like SSE instead of low level constructs, will make it easy to understand. ain't SSE a low level construct too? What about another approach for the cpuid issue: I think that dealing with specific flags is pretty error prone on all levels - virt-mgr, libvirt, qemu, migration, and even the guest. We can't just 'invent' new cpu modules that will be a combination of flags we have on the host. It might work for some kernels/apps but it might break others. In addition to cpuid we have the stepping and more MSRs that needed to be hidden/exposed. The offer: Libvirt, virt-mgr, qemu will not deal with lowlevel bits of the cpuid. We'll use predefined cpu modules to be emulated. These predefined modules will represent a real module that was once shipped by Intel/AMD. We'll write an additional tool, not under libvirt, that will be able to calculate all the virtual cpus that can run over the physical cpu. This tool will be the one messing with /proc/cpuinfo, etc. Example (theoretical): Physical cpu is Intel(R) Core(TM)2 Duo CPU T9600 @ 2.80GHz the output of the tool will be: shellcomputeVirtCpuCapabilities core2duo core solo 486 pentium3 .. Libvirt will only expose the real physical cpu and all of the outputs above. Higher level mgmt will compute the best -cpu to pick for the VM, and it will take account the user needs for performance or for migration flexibility. The cons: - Simple for all levels - Fast implementation - Accurate representative of real cpus The pros: - none - we should write the tool anyway - Maybe we hide some of the possibilities, but as said above, it is safer and friendlier. I also recommend of adding a field to be added next to the cpu for flexibility and possible future changes + dealing with problematic bios with NX bit disabled. Regards, Dor Thanks Mukesh On Thu, Sep 3, 2009 at 3:09 PM, Daniel P. Berrangeberra...@redhat.com wrote: On Thu, Sep 03, 2009 at 11:19:47AM +0300, Dor Laor wrote: On 09/02/2009 07:09 PM, Daniel P. Berrange wrote: On Wed, Sep 02, 2009 at 11:59:39AM -0400, Jim Paris wrote: Jiri Denemark wrote: Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. My proposal is to define new 'cpuid' feature advertised in guest capabilities: ... domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask ... What are your opinions about this? I think it's too low-level, and the structure is x86-specific. QEMU and KVM compute their CPUID response based on arguments to the -cpu argument, e.g.: -cpu core2duo,model=23,+ssse3,+lahf_lm I think a similar structure makes more sense for libvirt, where the configuration generally avoids big blocks of binary data, and the XML format should suit other architectures as well. I'm going backforth on this too. We essentially have 3 options - Named CPU + flags/features - CPUID masks - Allow either If we do either of the first two, we have to translate between the two formats for one or more of the hypervisors. For the last one we are just punting the problem off to applications. If we choose CPUID, and made QEMU driver convert to named CPU + flags we'd be stuck for non-x86 as you say. Why is that? cpu model + flags may apply for other arch too. If we have CPUID in the XML, there is no meaningful CPUID register representation for sparc/ppc/arm/etc. It is an x86 concept, which is almost certainly why QEMU uses named CPU models + named flags instead of CPUID as is public facing config. Xen/VMWare of course don't have this limitation since they only really care about x86. So really QEMU's CPU model + flags approach is more generic albeit being much more verbose to achieve the same level of expressivity. If we chose named CPU + flags, and made VMWare/Xen convert to raw CPUID we'd potentially loose information if user had defined a config with a raw CPUID mask outside context of libvirt. The other thing to remember is that CPUID also encodes sockets/cores/ threads topology data, and it'd be very desirable to expose that in a sensible fashion (ie not a bitmask). On balance i'm currently leaning to named CPU + flags + expliciti topology data because although its harder to implement for Xen/VMWare I think its much nicer to applicationsusers. We might loose a tiny bit of data in the CPU -named/flags conversion for Xen/VMWare but I reckon we can get it good enough that most people won't really care about that. Daniel There are 2 more issues to consider: 1. The VMW approach with all the cpuid bits might be ok, the problem is to map it into qemu model, will libvirt to that? THe
Re: [libvirt] [RFC] Support for CPUID masking
Speaking from an x86 angle,providing an ability to enable or disable high level constructs like SSE instead of low level constructs, will make it easy to understand. Thanks Mukesh On Thu, Sep 3, 2009 at 3:09 PM, Daniel P. Berrange berra...@redhat.com wrote: On Thu, Sep 03, 2009 at 11:19:47AM +0300, Dor Laor wrote: On 09/02/2009 07:09 PM, Daniel P. Berrange wrote: On Wed, Sep 02, 2009 at 11:59:39AM -0400, Jim Paris wrote: Jiri Denemark wrote: Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. My proposal is to define new 'cpuid' feature advertised in guest capabilities: ... domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask ... What are your opinions about this? I think it's too low-level, and the structure is x86-specific. QEMU and KVM compute their CPUID response based on arguments to the -cpu argument, e.g.: -cpu core2duo,model=23,+ssse3,+lahf_lm I think a similar structure makes more sense for libvirt, where the configuration generally avoids big blocks of binary data, and the XML format should suit other architectures as well. I'm going back forth on this too. We essentially have 3 options - Named CPU + flags/features - CPUID masks - Allow either If we do either of the first two, we have to translate between the two formats for one or more of the hypervisors. For the last one we are just punting the problem off to applications. If we choose CPUID, and made QEMU driver convert to named CPU + flags we'd be stuck for non-x86 as you say. Why is that? cpu model + flags may apply for other arch too. If we have CPUID in the XML, there is no meaningful CPUID register representation for sparc/ppc/arm/etc. It is an x86 concept, which is almost certainly why QEMU uses named CPU models + named flags instead of CPUID as is public facing config. Xen/VMWare of course don't have this limitation since they only really care about x86. So really QEMU's CPU model + flags approach is more generic albeit being much more verbose to achieve the same level of expressivity. If we chose named CPU + flags, and made VMWare/Xen convert to raw CPUID we'd potentially loose information if user had defined a config with a raw CPUID mask outside context of libvirt. The other thing to remember is that CPUID also encodes sockets/cores/ threads topology data, and it'd be very desirable to expose that in a sensible fashion (ie not a bitmask). On balance i'm currently leaning to named CPU + flags + expliciti topology data because although its harder to implement for Xen/VMWare I think its much nicer to applications users. We might loose a tiny bit of data in the CPU - named/flags conversion for Xen/VMWare but I reckon we can get it good enough that most people won't really care about that. Daniel There are 2 more issues to consider: 1. The VMW approach with all the cpuid bits might be ok, the problem is to map it into qemu model, will libvirt to that? THe problem is that CPUID is not viable for non-x86 archs so can't really be used as our master representation 2. If we use the qemu approach, the host information (cpuids) need to travel to higher mgmt level in order to allow computation of greatest common denominator. Yes, whatever we decide for exposing guest CPU model/flags/etc should be equally applied to the libvirt capabilities XML so that apps can query physical host data Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking v2
On Fri, Sep 04, 2009 at 04:58:25PM +0200, Jiri Denemark wrote: Firstly, CPU topology and all (actually all that libvirt knows about) CPU features have to be advertised in host capabilities: host cpu ... features featureNAME/feature /features topology socketsNUMBER_OF_SOCKETS/sockets coresCORES_PER_SOCKET/cores threadsTHREADS_PER_CORE/threads /topology /cpu ... /host FWIW, we already have the host topology sockets/core/threads exposed in the virNodeInfo API / struct, though I don't see any harm in having it in the node capabilities XML too, particularly since we put NUMA topology in there. I'm not 100% sure we should represent CPU features as featureNAME/feature especially because some features are currently advertised as NAME/. However, extending XML schema every time a new feature is introduced doesn't look like a good idea at all. The problem is we can't get rid of NAME/-style features, which would result in redundancy: features vmx/ featurevmx/feature /features But I think it's better than changing the schema to add new features. I think we need more than just the features in the capabilties XML though. eg, if an application wants to configure a guest with a CPU model of 'core2duo' then it needs to know whether the host OS is at least a 'core2duo' or a superset. In essence I think the host capabilities XML needs to be more closely aligned with your proposed guest XML, specifically including a base CPU model name, along with any additional features beyond the basic set provided by that model. Which brings me neatly to the next question The host capabilities XML for some random machine says the host CPU is a 'core2duo' + 'ssse3' + '3dnow'. There is a guest to be run with a XML config requesting 'pentium3' + 'ssse3' as a minimum requirement. Now pretend you are not a human who knows pentium3 is a sub-set of core2duo. How do we know whether it is possible to run the guest on that host ? We could say that we'll make 'virDomainCreate' just throw an error when you try to start a guest (or incoming migration, etc), but if we have a data center of many hosts, apps won't want to just try to start a guest on each host. They'll want some way to figure out equivalence between CPU + feature sets. Perhaps this suggests we want a virConnectCompareCPU(conn, guest cpu xml fragment) which returns 0 if the CPU is not compatible (ie subset), 1 if it is identical, or 2 if it is a superset. If we further declare that host capabilities for CPU model follow the same schema a guest XML for CPU model, we can use this same API to test 2 separate hosts for equivalence and thus figure out the lowest common denominator between a set of hosts also thus what guests are available for that set of hosts. For x86, this would require libvirt internal driver to have a xml - cpuid convertor, but then we already need one of those if we've to implement this stuff for Xen and VMWare drivers so I don't see this as too bad. We also of course need a cpuid - xml convertor to populate the host capabilities XML. For all this I'm thining we should have some basic external data files which map named CPUs to sets of CPUID features, and named flags to CPUID bits. Populate this with theset of CPUs QEMU knows about for now, and then we can extend this later simply by dropping in new data files. Back to your question about duplication: features vmx/ featurevmx/feature /features Just ignore the fact that we have vmx, pae + svm features defined for now. Focus on determining what XML schema we want to use consistently across host + guest for describing a CPU model + features. Once that's determined, we'll just fill in the legacy vmx/pae/svm features based off the data for the new format and recommend in the docs not to use the old style. Secondly, drivers which support detailed CPU specification have to advertise it in guest capabilities. In case features are meant to be hypervisor features, than it could look like: guest ... features cpu/ /features /guest But if they are meant to be CPU features, we need to come up with something else: guest ... cpu_selection/ /guest I'm not sure how to deal with named CPUs suggested by Dan. Either we need to come up with global set of named CPUs and document what they mean or let drivers specify their own named CPUs and advertise them through guest capabilities: guest ... cpu model=NAME featureNAME/feature ... /cpu /guest The former approach would make matching named CPUs with those defined by a hypervisor (such as qemu) quite hard. The latter could bring the need for hardcoding features provided
Re: [libvirt] [RFC] Support for CPUID masking v2
[ Sending again as my mail from yesterday seems to not have gone out :-( ] On Fri, Sep 04, 2009 at 04:58:25PM +0200, Jiri Denemark wrote: Hi, This is an attempt to provide similar flexibility to CPU ID masking without being x86-specific and unfriendly to users. As suggested by Dan, we need a way to specify both CPU flags and topology to achieve this goal. Right, thanks for trying to get this rolling :-) Firstly, CPU topology and all (actually all that libvirt knows about) CPU features have to be advertised in host capabilities: host cpu ... features featureNAME/feature /features topology socketsNUMBER_OF_SOCKETS/sockets coresCORES_PER_SOCKET/cores threadsTHREADS_PER_CORE/threads /topology topology sockets=x cores=y threads=z/ would work too and give the possibility to extend in a completely different way later by using subelement if CPU architeture were to evolve drastically later. /cpu ... /host I'm not 100% sure we should represent CPU features as featureNAME/feature especially because some features are currently advertised as NAME/. However, extending XML schema every time a new feature is introduced doesn't look like a good idea at all. The problem is we can't get rid of NAME/-style features, which would result in redundancy: features vmx/ featurevmx/feature /features I'm not afraid of that, it's not ideal but since those are virtualization related features having them separated sounds fine. We just can't grow the schemas and parsing code to accomodate a different element for each different name. IMHO the worst is that the definition of the names. First there is gonna be a bunch of them and second their name if you rely just on the procinfo output may not be sufficient in the absolute. Registries are an nightmare by definition, and we should not add a registry of features in libvirt, nor try to assert any semantic to those names. So I'm afraid we are good for just sampling/dumping /proc/cpuinfo and leave the mess to the kernel. The feature list will grow quite long but that's fine IMHO. But I think it's better than changing the schema to add new features. Yeah that's unmaintainable. Secondly, drivers which support detailed CPU specification have to advertise it in guest capabilities. In case features are meant to be hypervisor features, than it could look like: guest ... features cpu/ /features /guest Somehow we will get the same mess, I assume QEmu interface can provide that list, right ? I'm also wondering if it's not possibly dependant on the machine, I hope not, i.e. that the emulated CPU features are not also dependent on the emaulated hardware... But if they are meant to be CPU features, we need to come up with something else: guest ... cpu_selection/ /guest Something like guest cpu model=foo featuresfpu vme de pse tsc msr pae mce cx8 apic/features /cpu cpu model=bar featuresfpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca/features /cpu /guest hoping it doesn't go per machine ! I'm not sure how to deal with named CPUs suggested by Dan. Either we need to come up with global set of named CPUs and document what they mean or let drivers specify their own named CPUs and advertise them through guest capabilities: guest ... cpu model=NAME featureNAME/feature ... /cpu /guest Again I would not build the registry in libvirt itself, at least as a first approach, let the drivers provide them if available, expose them in the capabilities for the given guest type. If we really start to see duplication, then maygbe we can provide an helper. We could certainly provide utilities APIs to extract the set of flags and topology informations from utils/ but I would let the drivers being repsonsible for the list in the end. The former approach would make matching named CPUs with those defined by a hypervisor (such as qemu) quite hard. The latter could bring the need for hardcoding features provided by specific CPU models or, in case we decide not to provide a list of features for each CPU model, it can complicate transferring a domain from one hypervisor to another. And finally, CPU may be configured in domain XML configuration: domain ... cpu model=NAME topology socketsNUMBER_OF_SOCKETS/sockets coresCORES_PER_SOCKET/cores threadsTHREADS_PER_CORE/threads /topology topology sockets=x cores=y threads=z/ Might be better, in any case it should be kept consistant with the capabilities section format. feature name=NAME mode=set|check
Re: [libvirt] [RFC] Support for CPUID masking v2
On Tue, Sep 22, 2009 at 02:41:08PM +0200, Daniel Veillard wrote: I'm not 100% sure we should represent CPU features as featureNAME/feature especially because some features are currently advertised as NAME/. However, extending XML schema every time a new feature is introduced doesn't look like a good idea at all. The problem is we can't get rid of NAME/-style features, which would result in redundancy: features vmx/ featurevmx/feature /features I'm not afraid of that, it's not ideal but since those are virtualization related features having them separated sounds fine. We just can't grow the schemas and parsing code to accomodate a different element for each different name. IMHO the worst is that the definition of the names. First there is gonna be a bunch of them and second their name if you rely just on the procinfo output may not be sufficient in the absolute. No, we should't rely on /proc/cpuinfo because that is Linux specific. For Xen and VMWare drivers we want a naming scheme for flags that is OS agnostic, in particular so Xen works on Solaris and VMWare works nicely on Windows. That's why I think we should define a naming scheme for all flags in libvirt, albeit not hardcoded in the source, or XML schema, but in an external data file that can be easily extended when new CPUs come out. Registries are an nightmare by definition, and we should not add a registry of features in libvirt, nor try to assert any semantic to those names. So I'm afraid we are good for just sampling/dumping /proc/cpuinfo and leave the mess to the kernel. The feature list will grow quite long but that's fine IMHO. We can actually keep the feature list very short. The key is that we expose a named CPU model which covers 95% of the host PCU features. We then justneed to list CPU features which are not explicitly part of that CPU moidel - which should be a mere handful - certainly less than 10. Secondly, drivers which support detailed CPU specification have to advertise it in guest capabilities. In case features are meant to be hypervisor features, than it could look like: guest ... features cpu/ /features /guest Somehow we will get the same mess, I assume QEmu interface can provide that list, right ? I'm also wondering if it's not possibly dependant on the machine, I hope not, i.e. that the emulated CPU features are not also dependent on the emaulated hardware... CPU features are really just a artifact of the CPU model. The existing 'acpi' and 'apic' flags are not CPU features - they are chipset features, so out of scope for this discussion But if they are meant to be CPU features, we need to come up with something else: guest ... cpu_selection/ /guest Something like guest cpu model=foo featuresfpu vme de pse tsc msr pae mce cx8 apic/features /cpu cpu model=bar featuresfpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca/features /cpu /guest hoping it doesn't go per machine ! Exposing flags as a big string like this is really not nice for applications. Having different representations for capabilities XML description of a CPU vs the guest XML description of a CPU is also undesirable, thus I think thta the capabilities XML should essentially follow whatever schema we decide to use for the guest XML Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking v2
On Tue, Sep 22, 2009 at 02:25:54PM +0100, Daniel P. Berrange wrote: On Tue, Sep 22, 2009 at 02:41:08PM +0200, Daniel Veillard wrote: IMHO the worst is that the definition of the names. First there is gonna be a bunch of them and second their name if you rely just on the procinfo output may not be sufficient in the absolute. No, we should't rely on /proc/cpuinfo because that is Linux specific. For Xen and VMWare drivers we want a naming scheme for flags that is OS agnostic, in particular so Xen works on Solaris and VMWare works nicely on Windows. For VMWare I expect the flag list and CPU descriptions to come from the driver, which can probably extract them from ESX itself. That's why I think we should define a naming scheme for all flags in libvirt, albeit not hardcoded in the source, or XML schema, but in an external data file that can be easily extended when new CPUs come out. I don't see how that's gonna scale. Just with the set of processors suppoorted by QEmu and the number of flags they may each export or not. Sure an external file would make maintainance way easier, but still... Registries are an nightmare by definition, and we should not add a registry of features in libvirt, nor try to assert any semantic to those names. So I'm afraid we are good for just sampling/dumping /proc/cpuinfo and leave the mess to the kernel. The feature list will grow quite long but that's fine IMHO. We can actually keep the feature list very short. The key is that we expose a named CPU model which covers 95% of the host PCU features. We then justneed to list CPU features which are not explicitly part of that CPU moidel - which should be a mere handful - certainly less than 10. how do you know those lists and subsets, and how are you gonna keep them on the long term. If you take a processor definition from 5 years ago and want to make sure none of the new CPU features are not used what's the scenario in practice ? Would the application hagve to know the logic behind the name we would be defining for the processor type ? Would it have to have that knowledge to know that based on that processor type then such and such flags are not set ? If we export names which are managed by libvirt, then it becomes libvirt responsability to define the matrix of flags names and their semantic. And that's really something I'm afraid of. I prefer to delegate to the kernel or virtualization layers (via the drivers) to provide those flags and semantics, ultimately they end up being maintained either by the chip founders themselves or the hypervisors implementors (VMWare). Secondly, drivers which support detailed CPU specification have to advertise it in guest capabilities. In case features are meant to be hypervisor features, than it could look like: guest ... features cpu/ /features /guest Somehow we will get the same mess, I assume QEmu interface can provide that list, right ? I'm also wondering if it's not possibly dependant on the machine, I hope not, i.e. that the emulated CPU features are not also dependent on the emaulated hardware... CPU features are really just a artifact of the CPU model. Hopefully ... but my experience with embedded (some time ago admitedly) made clear that such or such processor feature may be activated or not depending on how they got wired. Even nowadays your CPU may have support for things which gets desactivated by the BIOS for example. Not that simple IMHO. The existing 'acpi' and 'apic' flags are not CPU features - they are chipset features, so out of scope for this discussion Okay Something like guest cpu model=foo featuresfpu vme de pse tsc msr pae mce cx8 apic/features /cpu cpu model=bar featuresfpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca/features /cpu /guest hoping it doesn't go per machine ! Exposing flags as a big string like this is really not nice for applications. Having different representations for capabilities XML description of a CPU vs the guest XML description of a CPU is also undesirable, thus I think thta the capabilities XML should essentially follow whatever schema we decide to use for the guest XML yes I agree we should unify both formats. The flag list as a single string isn't fun for applications, granted, something more structured would be better. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ dan...@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/ -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking v2
On Tue, Sep 22, 2009 at 03:52:08PM +0200, Daniel Veillard wrote: On Tue, Sep 22, 2009 at 02:25:54PM +0100, Daniel P. Berrange wrote: No, we should't rely on /proc/cpuinfo because that is Linux specific. For Xen and VMWare drivers we want a naming scheme for flags that is OS agnostic, in particular so Xen works on Solaris and VMWare works nicely on Windows. For VMWare I expect the flag list and CPU descriptions to come from the driver, which can probably extract them from ESX itself. VMWare doesnt expose any named flags / CPU. It just exports the raw CPUID bitmask. So libvirt has to maintain a database of named + flags to convert the VMWare CPUID into something useful. The same situation exists with Xen. That's why I think we should define a naming scheme for all flags in libvirt, albeit not hardcoded in the source, or XML schema, but in an external data file that can be easily extended when new CPUs come out. I don't see how that's gonna scale. Just with the set of processors suppoorted by QEmu and the number of flags they may each export or not. Sure an external file would make maintainance way easier, but still... The key is that you don't try to create named CPU model for every possible CPU that Intel/AMD release. You just have a handful of CPU models, and then uses flags to indicate extra features. Thus it becomes tradeoff between the number of CPU models available, vs number of extra flags an app has to list which lets us control the way it scales while still giving flexibility to apps. As a point of reference, QEMU has 10 named CPU models currently for x86, but with the combination of possible names + flags, it can expose many 100's of different CPUs to the guest. Registries are an nightmare by definition, and we should not add a registry of features in libvirt, nor try to assert any semantic to those names. So I'm afraid we are good for just sampling/dumping /proc/cpuinfo and leave the mess to the kernel. The feature list will grow quite long but that's fine IMHO. We can actually keep the feature list very short. The key is that we expose a named CPU model which covers 95% of the host PCU features. We then justneed to list CPU features which are not explicitly part of that CPU moidel - which should be a mere handful - certainly less than 10. how do you know those lists and subsets, and how are you gonna keep them on the long term. If you take a processor definition from 5 years ago and want to make sure none of the new CPU features are not used what's the scenario in practice ? Would the application hagve to know the logic behind the name we would be defining for the processor type ? Would it have to have that knowledge to know that based on that processor type then such and such flags are not set ? If we export names which are managed by libvirt, then it becomes libvirt responsability to define the matrix of flags names and their semantic. And that's really something I'm afraid of. I prefer to delegate to the kernel or virtualization layers (via the drivers) to provide those flags and semantics, ultimately they end up being maintained either by the chip founders themselves or the hypervisors implementors (VMWare). The key issue here is there is nothing to delegate to in VMWare or Xen case, since both use the raw CPUID format as their config model - which is x86 specific so we need to apply a conversion in libvirt. Once we're doing that, it becomes trivial todo it for exposing the host CPU model too. Somehow we will get the same mess, I assume QEmu interface can provide that list, right ? I'm also wondering if it's not possibly dependant on the machine, I hope not, i.e. that the emulated CPU features are not also dependent on the emaulated hardware... CPU features are really just a artifact of the CPU model. Hopefully ... but my experience with embedded (some time ago admitedly) made clear that such or such processor feature may be activated or not depending on how they got wired. Even nowadays your CPU may have support for things which gets desactivated by the BIOS for example. Not that simple IMHO. The BIOS settings aren't actually toggling the CPU features. If you have VT/SVM disabled in the BIOS, it'll still be visible in the /proc/cpuinfo flags data. The BIOS is toggling something else to prevent the feature being used, even when present. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking v2
On Tue, Sep 22, 2009 at 03:01:18PM +0100, Daniel P. Berrange wrote: On Tue, Sep 22, 2009 at 03:52:08PM +0200, Daniel Veillard wrote: On Tue, Sep 22, 2009 at 02:25:54PM +0100, Daniel P. Berrange wrote: No, we should't rely on /proc/cpuinfo because that is Linux specific. For Xen and VMWare drivers we want a naming scheme for flags that is OS agnostic, in particular so Xen works on Solaris and VMWare works nicely on Windows. For VMWare I expect the flag list and CPU descriptions to come from the driver, which can probably extract them from ESX itself. VMWare doesnt expose any named flags / CPU. It just exports the raw CPUID bitmask. So libvirt has to maintain a database of named + flags to convert the VMWare CPUID into something useful. The same situation exists with Xen. sigh/ That's why I think we should define a naming scheme for all flags in libvirt, albeit not hardcoded in the source, or XML schema, but in an external data file that can be easily extended when new CPUs come out. I don't see how that's gonna scale. Just with the set of processors suppoorted by QEmu and the number of flags they may each export or not. Sure an external file would make maintainance way easier, but still... The key is that you don't try to create named CPU model for every possible CPU that Intel/AMD release. You just have a handful of CPU models, and then uses flags to indicate extra features. Thus it becomes tradeoff between the number of CPU models available, vs number of extra flags an app has to list which lets us control the way it scales while still giving flexibility to apps. As a point of reference, QEMU has 10 named CPU models currently for x86, but with the combination of possible names + flags, it can expose many 100's of different CPUs to the guest. Okay, maybe we can keep this maintainable. The alternative is a very unfriendly and unreliable API. It's just a shame that those things end up in libvirt, because honnestly it really sounds like low level logic that not directly tied to virtualization and which should really come from the system, but since we have remote only drivers like VMWare, that doesn't exist and we would need this portable then okay. If we go that way, then yes definitely let's make those descriptions an external file easilly updated by the distro or sysadmin, so that we don't get upstream update request in 5 years when nobody knows anymore what a core2duo might have been... Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ dan...@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/ -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking v2
I'm not 100% sure we should represent CPU features as featureNAME/feature especially because some features are currently advertised as NAME/. However, extending XML schema every time a new feature is introduced doesn't look like a good idea at all. The problem is we can't get rid of NAME/-style features, which would result in redundancy: features vmx/ featurevmx/feature /features But I think it's better than changing the schema to add new features. I think we need more than just the features in the capabilties XML though. eg, if an application wants to configure a guest with a CPU model of 'core2duo' then it needs to know whether the host OS is at least a 'core2duo' or a superset. In essence I think the host capabilities XML needs to be more closely aligned with your proposed guest XML, specifically including a base CPU model name, along with any additional features beyond the basic set provided by that model. Which brings me neatly to the next question The host capabilities XML for some random machine says the host CPU is a 'core2duo' + 'ssse3' + '3dnow'. There is a guest to be run with a XML config requesting 'pentium3' + 'ssse3' as a minimum requirement. Now pretend you are not a human who knows pentium3 is a sub-set of core2duo. How do we know whether it is possible to run the guest on that host ? When I was proposing this, I thought about CPU name to be just a shortcut to a set of features. That is, you could see if pentium3 is a subset of core2duo by just translating them into a list of features and comparing them. Thanks for your comments. Jirka -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking v2
On Tue, Sep 22, 2009 at 05:51:02PM +0200, Jiri Denemark wrote: I'm not 100% sure we should represent CPU features as featureNAME/feature especially because some features are currently advertised as NAME/. However, extending XML schema every time a new feature is introduced doesn't look like a good idea at all. The problem is we can't get rid of NAME/-style features, which would result in redundancy: features vmx/ featurevmx/feature /features But I think it's better than changing the schema to add new features. I think we need more than just the features in the capabilties XML though. eg, if an application wants to configure a guest with a CPU model of 'core2duo' then it needs to know whether the host OS is at least a 'core2duo' or a superset. In essence I think the host capabilities XML needs to be more closely aligned with your proposed guest XML, specifically including a base CPU model name, along with any additional features beyond the basic set provided by that model. Which brings me neatly to the next question The host capabilities XML for some random machine says the host CPU is a 'core2duo' + 'ssse3' + '3dnow'. There is a guest to be run with a XML config requesting 'pentium3' + 'ssse3' as a minimum requirement. Now pretend you are not a human who knows pentium3 is a sub-set of core2duo. How do we know whether it is possible to run the guest on that host ? When I was proposing this, I thought about CPU name to be just a shortcut to a set of features. That is, you could see if pentium3 is a subset of core2duo by just translating them into a list of features and comparing them. True, but the issue with that is that it is an x86 specific concept. non-x86 CPU models can't be decomposed into a list of features for comparison. So I reckon its best to provide some explicit API or facility in libvirt to compare 2 CPU+feature descriptions for compatability, so we can hide the x86 specific bits from applications Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking v2
I'm not sure how to deal with named CPUs suggested by Dan. Either we need to come up with global set of named CPUs and document what they mean or let drivers specify their own named CPUs and advertise them through guest capabilities: guest ... cpu model=NAME featureNAME/feature ... /cpu /guest [IH] you also need to support removing a feature from the base cpu model, if it is disabled by bios, like the nx flag). Indeed, the above XML snippet describes capabilities, that is what features are turned on by each model name. ... And finally, CPU may be configured in domain XML configuration: domain ... cpu model=NAME topology socketsNUMBER_OF_SOCKETS/sockets coresCORES_PER_SOCKET/cores threadsTHREADS_PER_CORE/threads /topology feature name=NAME mode=set|check value=on|off/ /cpu /domain Mode 'check' checks physical CPU for the feature and refuses the domain to start if it doesn't match. VCPU feature is set to the same value. Mode 'set' just sets the VCPU feature. While here, when configuring a domain, you would use something like cpu model=whatever feature name=sse6 mode=set value=off/ /cpu to turn off 'sse6' feature which was turned on by selecting CPU model 'whatever'. Jirka -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
RE: [libvirt] [RFC] Support for CPUID masking v2
From: libvir-list-boun...@redhat.com [mailto:libvir-list- Hi, This is an attempt to provide similar flexibility to CPU ID masking without being x86-specific and unfriendly to users. As suggested by Dan, we need a way to specify both CPU flags and topology to achieve this goal. Firstly, CPU topology and all (actually all that libvirt knows about) CPU features have to be advertised in host capabilities: host cpu ... features featureNAME/feature /features topology socketsNUMBER_OF_SOCKETS/sockets coresCORES_PER_SOCKET/cores threadsTHREADS_PER_CORE/threads /topology /cpu ... /host I'm not 100% sure we should represent CPU features as featureNAME/feature especially because some features are currently advertised as NAME/. However, extending XML schema every time a new feature is introduced doesn't look like a good idea at all. The problem is we can't get rid of NAME/-style features, which would result in redundancy: features vmx/ featurevmx/feature /features But I think it's better than changing the schema to add new features. Secondly, drivers which support detailed CPU specification have to advertise it in guest capabilities. In case features are meant to be hypervisor features, than it could look like: guest ... features cpu/ /features /guest But if they are meant to be CPU features, we need to come up with something else: guest ... cpu_selection/ /guest I'm not sure how to deal with named CPUs suggested by Dan. Either we need to come up with global set of named CPUs and document what they mean or let drivers specify their own named CPUs and advertise them through guest capabilities: guest ... cpu model=NAME featureNAME/feature ... /cpu /guest [IH] you also need to support removing a feature from the base cpu model, if it is disabled by bios, like the nx flag). The former approach would make matching named CPUs with those defined by a hypervisor (such as qemu) quite hard. The latter could bring the need for hardcoding features provided by specific CPU models or, in case we decide not to provide a list of features for each CPU model, it can complicate transferring a domain from one hypervisor to another. And finally, CPU may be configured in domain XML configuration: domain ... cpu model=NAME topology socketsNUMBER_OF_SOCKETS/sockets coresCORES_PER_SOCKET/cores threadsTHREADS_PER_CORE/threads /topology feature name=NAME mode=set|check value=on|off/ /cpu /domain Mode 'check' checks physical CPU for the feature and refuses the domain to start if it doesn't match. VCPU feature is set to the same value. Mode 'set' just sets the VCPU feature. Final note: topology could also be called cpu_topology to avoid confusion with NUMA topology, which is used in host capabilities. However, I prefer cputopology.../topology/cpu over cpucpu_topology.../cpu_topology/cpu. Thanks for your comments. Jirka -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking
On 09/02/2009 07:09 PM, Daniel P. Berrange wrote: On Wed, Sep 02, 2009 at 11:59:39AM -0400, Jim Paris wrote: Jiri Denemark wrote: Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. My proposal is to define new 'cpuid' feature advertised in guest capabilities: ... domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask ... What are your opinions about this? I think it's too low-level, and the structure is x86-specific. QEMU and KVM compute their CPUID response based on arguments to the -cpu argument, e.g.: -cpu core2duo,model=23,+ssse3,+lahf_lm I think a similar structure makes more sense for libvirt, where the configuration generally avoids big blocks of binary data, and the XML format should suit other architectures as well. I'm going back forth on this too. We essentially have 3 options - Named CPU + flags/features - CPUID masks - Allow either If we do either of the first two, we have to translate between the two formats for one or more of the hypervisors. For the last one we are just punting the problem off to applications. If we choose CPUID, and made QEMU driver convert to named CPU + flags we'd be stuck for non-x86 as you say. Why is that? cpu model + flags may apply for other arch too. If we chose named CPU + flags, and made VMWare/Xen convert to raw CPUID we'd potentially loose information if user had defined a config with a raw CPUID mask outside context of libvirt. The other thing to remember is that CPUID also encodes sockets/cores/ threads topology data, and it'd be very desirable to expose that in a sensible fashion (ie not a bitmask). On balance i'm currently leaning to named CPU + flags + expliciti topology data because although its harder to implement for Xen/VMWare I think its much nicer to applications users. We might loose a tiny bit of data in the CPU - named/flags conversion for Xen/VMWare but I reckon we can get it good enough that most people won't really care about that. Daniel There are 2 more issues to consider: 1. The VMW approach with all the cpuid bits might be ok, the problem is to map it into qemu model, will libvirt to that? 2. If we use the qemu approach, the host information (cpuids) need to travel to higher mgmt level in order to allow computation of greatest common denominator. Regards, Dor -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[libvirt] [RFC] Support for CPUID masking v2
Hi, This is an attempt to provide similar flexibility to CPU ID masking without being x86-specific and unfriendly to users. As suggested by Dan, we need a way to specify both CPU flags and topology to achieve this goal. Firstly, CPU topology and all (actually all that libvirt knows about) CPU features have to be advertised in host capabilities: host cpu ... features featureNAME/feature /features topology socketsNUMBER_OF_SOCKETS/sockets coresCORES_PER_SOCKET/cores threadsTHREADS_PER_CORE/threads /topology /cpu ... /host I'm not 100% sure we should represent CPU features as featureNAME/feature especially because some features are currently advertised as NAME/. However, extending XML schema every time a new feature is introduced doesn't look like a good idea at all. The problem is we can't get rid of NAME/-style features, which would result in redundancy: features vmx/ featurevmx/feature /features But I think it's better than changing the schema to add new features. Secondly, drivers which support detailed CPU specification have to advertise it in guest capabilities. In case features are meant to be hypervisor features, than it could look like: guest ... features cpu/ /features /guest But if they are meant to be CPU features, we need to come up with something else: guest ... cpu_selection/ /guest I'm not sure how to deal with named CPUs suggested by Dan. Either we need to come up with global set of named CPUs and document what they mean or let drivers specify their own named CPUs and advertise them through guest capabilities: guest ... cpu model=NAME featureNAME/feature ... /cpu /guest The former approach would make matching named CPUs with those defined by a hypervisor (such as qemu) quite hard. The latter could bring the need for hardcoding features provided by specific CPU models or, in case we decide not to provide a list of features for each CPU model, it can complicate transferring a domain from one hypervisor to another. And finally, CPU may be configured in domain XML configuration: domain ... cpu model=NAME topology socketsNUMBER_OF_SOCKETS/sockets coresCORES_PER_SOCKET/cores threadsTHREADS_PER_CORE/threads /topology feature name=NAME mode=set|check value=on|off/ /cpu /domain Mode 'check' checks physical CPU for the feature and refuses the domain to start if it doesn't match. VCPU feature is set to the same value. Mode 'set' just sets the VCPU feature. Final note: topology could also be called cpu_topology to avoid confusion with NUMA topology, which is used in host capabilities. However, I prefer cputopology.../topology/cpu over cpucpu_topology.../cpu_topology/cpu. Thanks for your comments. Jirka -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking
We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. My proposal is to define new 'cpuid' feature advertised in guest capabilities: ... domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask ... What are your opinions about this? I think it's too low-level, and the structure is x86-specific. Hmm, right, it's only for x86... I'll come up with something better. Jirka -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking
On Thu, Sep 03, 2009 at 11:19:47AM +0300, Dor Laor wrote: On 09/02/2009 07:09 PM, Daniel P. Berrange wrote: On Wed, Sep 02, 2009 at 11:59:39AM -0400, Jim Paris wrote: Jiri Denemark wrote: Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. My proposal is to define new 'cpuid' feature advertised in guest capabilities: ... domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask ... What are your opinions about this? I think it's too low-level, and the structure is x86-specific. QEMU and KVM compute their CPUID response based on arguments to the -cpu argument, e.g.: -cpu core2duo,model=23,+ssse3,+lahf_lm I think a similar structure makes more sense for libvirt, where the configuration generally avoids big blocks of binary data, and the XML format should suit other architectures as well. I'm going back forth on this too. We essentially have 3 options - Named CPU + flags/features - CPUID masks - Allow either If we do either of the first two, we have to translate between the two formats for one or more of the hypervisors. For the last one we are just punting the problem off to applications. If we choose CPUID, and made QEMU driver convert to named CPU + flags we'd be stuck for non-x86 as you say. Why is that? cpu model + flags may apply for other arch too. If we have CPUID in the XML, there is no meaningful CPUID register representation for sparc/ppc/arm/etc. It is an x86 concept, which is almost certainly why QEMU uses named CPU models + named flags instead of CPUID as is public facing config. Xen/VMWare of course don't have this limitation since they only really care about x86. So really QEMU's CPU model + flags approach is more generic albeit being much more verbose to achieve the same level of expressivity. If we chose named CPU + flags, and made VMWare/Xen convert to raw CPUID we'd potentially loose information if user had defined a config with a raw CPUID mask outside context of libvirt. The other thing to remember is that CPUID also encodes sockets/cores/ threads topology data, and it'd be very desirable to expose that in a sensible fashion (ie not a bitmask). On balance i'm currently leaning to named CPU + flags + expliciti topology data because although its harder to implement for Xen/VMWare I think its much nicer to applications users. We might loose a tiny bit of data in the CPU - named/flags conversion for Xen/VMWare but I reckon we can get it good enough that most people won't really care about that. Daniel There are 2 more issues to consider: 1. The VMW approach with all the cpuid bits might be ok, the problem is to map it into qemu model, will libvirt to that? THe problem is that CPUID is not viable for non-x86 archs so can't really be used as our master representation 2. If we use the qemu approach, the host information (cpuids) need to travel to higher mgmt level in order to allow computation of greatest common denominator. Yes, whatever we decide for exposing guest CPU model/flags/etc should be equally applied to the libvirt capabilities XML so that apps can query physical host data Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[libvirt] [RFC] Support for CPUID masking
Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. My proposal is to define new 'cpuid' feature advertised in guest capabilities: guest ... features cpuid/ /feature /guest When a driver supports cpuid feature, one can use it to mask/check for specific bits returned by CPU ID as follows: domain ... features cpuid mask level='hex' register='eax|ebx|ecx|edx'MASK/mask /cpuid /features ... /domain Where - level is a hexadecimal number used as an input to CPU ID (i.e. eax register), - register is one of eax, ebx, ecx or edx, - and MASK is a string with the following format: ::::::: with m being 1-bit mask for the corresponding bit in the register. There are three possibilities of specifying what values can be used for 'm': - let it be driver-specific, - define all possible values, - define a common set of values and allow drivers to specify their own additional values. I think the third is the way to go as it lowers the confusion of different values used by different drivers for the same purpose while maintaining the flexibility to support driver-specific masks. The following could be a good set of predefined common values: - 1 force the bit to be 1 - 0 force the bit to be 0 - x don't care, i.e., use driver's default value - T require the bit to be 1 - F require the bit to be 0 Some examples of what it could look like follow: capabilities ... guest os_typexen/os_type ... features pae/ cpuid/ /features /guest ... /capabilities domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask mask level='1' register='ecx' ::::::xx1x: /mask mask level='1' register='edx' xxx1::::::: /mask mask level='8001' register='ecx' :::::::xx1x /mask mask level='8008' register='ecx' ::::::xx00:1001 /mask /cpuid /features ... /domain What are your opinions about this? Thank for all comments. Jirka -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking
2009/9/2 Jiri Denemark jdene...@redhat.com: Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. [...] domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask mask level='1' register='ecx' ::::::xx1x: /mask mask level='1' register='edx' xxx1::::::: /mask mask level='8001' register='ecx' :::::::xx1x /mask mask level='8008' register='ecx' ::::::xx00:1001 /mask /cpuid /features ... /domain I like the proposed mapping for the domain XML, because it's an 1:1 mapping of what VMware uses in the VI API [1] and the VMX config. Beside that VMware has two more possible values for the CPUID bits: H and R. Both are used to define how to handle/interpret those bits in the context of VMotion (migration). For example the domain XML snippet above maps to this VMX snippet: cpuid.1.ebx = 1010 cpuid.1.ecx = XX1X cpuid.1.edx = XXX1 cpuid.8001.ecx = XX1X cpuid.8008.ecx = XX001001 Matthias [1] http://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.host.CpuIdInfo.html -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking
Jiri Denemark wrote: Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. My proposal is to define new 'cpuid' feature advertised in guest capabilities: ... domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask ... What are your opinions about this? I think it's too low-level, and the structure is x86-specific. QEMU and KVM compute their CPUID response based on arguments to the -cpu argument, e.g.: -cpu core2duo,model=23,+ssse3,+lahf_lm I think a similar structure makes more sense for libvirt, where the configuration generally avoids big blocks of binary data, and the XML format should suit other architectures as well. -jim -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC] Support for CPUID masking
On Wed, Sep 02, 2009 at 11:59:39AM -0400, Jim Paris wrote: Jiri Denemark wrote: Hi, We need to provide support for CPU ID masking. Xen and VMware ESX are examples of current hypervisors which support such masking. My proposal is to define new 'cpuid' feature advertised in guest capabilities: ... domain type='xen' id='42' ... features pae/ acpi/ apic/ cpuid mask level='1' register='ebx' :::1010:::: /mask ... What are your opinions about this? I think it's too low-level, and the structure is x86-specific. QEMU and KVM compute their CPUID response based on arguments to the -cpu argument, e.g.: -cpu core2duo,model=23,+ssse3,+lahf_lm I think a similar structure makes more sense for libvirt, where the configuration generally avoids big blocks of binary data, and the XML format should suit other architectures as well. I'm going back forth on this too. We essentially have 3 options - Named CPU + flags/features - CPUID masks - Allow either If we do either of the first two, we have to translate between the two formats for one or more of the hypervisors. For the last one we are just punting the problem off to applications. If we choose CPUID, and made QEMU driver convert to named CPU + flags we'd be stuck for non-x86 as you say. If we chose named CPU + flags, and made VMWare/Xen convert to raw CPUID we'd potentially loose information if user had defined a config with a raw CPUID mask outside context of libvirt. The other thing to remember is that CPUID also encodes sockets/cores/ threads topology data, and it'd be very desirable to expose that in a sensible fashion (ie not a bitmask). On balance i'm currently leaning to named CPU + flags + expliciti topology data because although its harder to implement for Xen/VMWare I think its much nicer to applications users. We might loose a tiny bit of data in the CPU - named/flags conversion for Xen/VMWare but I reckon we can get it good enough that most people won't really care about that. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
