Re: [virt-tools-list] [libvirt] Project for profiles and defaults for libvirt domains
On Thu, Mar 22, 2018 at 09:40:40AM +0100, Pavel Hrdina wrote: > On Tue, Mar 20, 2018 at 03:10:12PM +, Daniel P. Berrangé wrote: > > > - I understand OpenStack has some really sensible and wisely chosen > > > and/or tested default values. > > > > In terms of default devices and OS specific choices, OpenStack's > > decisions have been largely inspired by previous work in oVirt > > and / or virt-manager. So there's obviously overlap in the > > conceptual area, but there's also plenty that is very specific > > to OpenStack - untangling the two extract the common bits from > > the app specific bits is hard. > > This would be handled by the application specific policies. The > virtuned will have some reasonable defaults that are known to work in > most cases and suits the majority of users, but it's clear that > sometimes you need some specific defaults and you would provide them > via the application policy. > > For example, to create a XML for windows guest the virtuned would not > probably select virtio devices because there are no drivers for them > in the standard windows installation, however, some management > application may have customized preinstalled disk images or customized > ISO images or it may be able to provide the drivers any other way, so > they would specify in the application policy that for windows guest > virtuned should use virtio as a default device model. As soon as we talk about configuring hardware specific to a guest OS, then that is in scope of existing libosinfo project, not something we should create a new project for. > This is probably the hardest part of creating higher level API on top > of libvirt, not every project may be willing to rewrite their existing > code. On the other hand, I know that for example Cockpit would benefit > from the virtuned providing this functionality via REST API. > > It's a chicken and egg problem, but if we can gather input from all the > existing projects that have their own implementation and figure out how > to make virtuned usable for all of them they might consider to start > using it. I don't doubt that Cockpit would like like, but based on previous efforts we've made I'm sceptical that anything beyond Cockpit would use any new API. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ virt-tools-list mailing list virt-tools-list@redhat.com https://www.redhat.com/mailman/listinfo/virt-tools-list
Re: [virt-tools-list] [libvirt] Project for profiles and defaults for libvirt domains
On Tue, Mar 20, 2018 at 03:10:12PM +, Daniel P. Berrangé wrote: > On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote: > > 1) Default devices/values > > > > Libvirt itself must default to whatever values there were before any > > particular element was introduced due to the fact that it strives to > > keep the guest ABI stable. That means, for example, that it can't just > > add -vmcoreinfo option (for KASLR support) or magically add the pvpanic > > device to all QEMU machines, even though it would be useful, as that > > would change the guest ABI. > > > > For default values this is even more obvious. Let's say someone figures > > out some "pretty good" default values for various HyperV enlightenment > > feature tunables. Libvirt can't magically change them, but each one of > > the projects building on top of it doesn't want to keep that list > > updated and take care of setting them in every new XML. Some projects > > don't even expose those to the end user as a knob, while others might. > > This gets very tricky, very fast. > > Lets say that you have an initial good set of hyperv config > tunables. Now sometime passes and it is decided that there is a > different, better set of config tunables. If the module that is > providing this policy to apps like OpenStack just updates itself > to provide this new policy, this can cause problems with the > existing deployed applications in a number of ways. > > First the new config probably depends on specific versions of > libvirt and QEMU, and you can't mandate to consuming apps which > versions they must be using. So you need a matrix of libvirt + > QEMU + config option settings. > > Even if you have the matching libvirt & QEMU versions, it is not > safe to assume the application will want to use the new policy. > An application may need live migration compatibility with older > versions. Or it may need to retain guaranteed ABI compatibility > with the way the VM was previously launched and be using transient > guests, generating the XML fresh each time. > > The application will have knowledge about when it wants to use new > vs old hyperv tunable policy, but exposing that to your policy module > is very tricky because it is inherantly application specific logic > largely determined by the way the application code is written. This is a good point and would be definitely an issue. One way how to solve it could be that the virtuned would be able to return you the default policy, the management application would save it somewhere if it cares about reproducing the same XML with the same default policies with newer version of virtuned/libvirt/qemu/kvm/kernel. The API to create a new domain or hotplugable device would take another optional parameter where you could pass the default policy in addition to application specific policy. That way you would get the same XML even with newer components with the same benefit that you wound not have to reimplement the logic. > > One more thing could be automatically figuring out best values based on > > libosinfo-provided data. > > > > 2) Policies > > > > Lot of the time there are parts of the domain definition that need to be > > added, but nobody really cares about them. Sometimes it's enough to > > have few templates, another time you might want to have a policy > > per-scenario and want to combine them in various ways. For example with > > the data provided by point 1). > > > > For example if you want PCI-Express, you need the q35 machine type, but > > you don't really want to care about the machine type. Or you want to > > use SPICE, but you don't want to care about adding QXL. > > > > What if some of these policies could be specified once (using some DSL > > for example), and used by virtuned to merge them in a unified and > > predictable way? > > > > 3) Abstracting the XML > > > > This is probably just usable for stateless apps, but it might happen > > that some apps don't really want to care about the XML at all. They > > just want an abstract view of the domain, possibly add/remove a device > > and that's it. We could do that as well. I can't really tell how much > > of a demand there is for it, though. > > It is safe to say that applications do not want to touch XML at all. > Any non-trivial application has created an abstraction around XML, > so that they have an API to express what they want, rather than > manipulating of strings to format/parse XML. > > The libvirt-gconfig project aims to provide a C API for manipulating > XML documents, with language bindings available via GObject > introspection so you can use it from Vala, Perl, Python, JavaScript, > etc. Go is notable missing, but for that we have libvirt-go-xml > which provides a set of native Go structs to represent the XML. > > The problem we've faced with libvirt-gconfig is that it is a really > hard sell to get applications to convert existing code to use it. > We've only had success where an applicaiton
Re: [virt-tools-list] [libvirt] Project for profiles and defaults for libvirt domains
On Wed, Mar 21, 2018 at 06:39:52PM +, Daniel P. Berrangé wrote: > On Wed, Mar 21, 2018 at 03:00:41PM -0300, Eduardo Habkost wrote: > > On Tue, Mar 20, 2018 at 03:10:12PM +, Daniel P. Berrangé wrote: > > > On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote: > > > > 1) Default devices/values > > > > > > > > Libvirt itself must default to whatever values there were before any > > > > particular element was introduced due to the fact that it strives to > > > > keep the guest ABI stable. That means, for example, that it can't just > > > > add -vmcoreinfo option (for KASLR support) or magically add the pvpanic > > > > device to all QEMU machines, even though it would be useful, as that > > > > would change the guest ABI. > > > > > > > > For default values this is even more obvious. Let's say someone figures > > > > out some "pretty good" default values for various HyperV enlightenment > > > > feature tunables. Libvirt can't magically change them, but each one of > > > > the projects building on top of it doesn't want to keep that list > > > > updated and take care of setting them in every new XML. Some projects > > > > don't even expose those to the end user as a knob, while others might. > > > > > > This gets very tricky, very fast. > > > > > > Lets say that you have an initial good set of hyperv config > > > tunables. Now sometime passes and it is decided that there is a > > > different, better set of config tunables. If the module that is > > > providing this policy to apps like OpenStack just updates itself > > > to provide this new policy, this can cause problems with the > > > existing deployed applications in a number of ways. > > > > > > First the new config probably depends on specific versions of > > > libvirt and QEMU, and you can't mandate to consuming apps which > > > versions they must be using. [...] > > > > This is true. > > > > > [...] So you need a matrix of libvirt + > > > QEMU + config option settings. > > > > But this is not. If config options need support on the lower > > levels of the stack (libvirt and/or QEMU and/or KVM and/or host > > hardware), it already has to be represented by libvirt host > > capabilities somehow, so management layers know it's available. > > > > This means any new config generation system can (and must) use > > host(s) capabilities as input before generating the > > configuration. > > I don't think it is that simple. The capabilities reflect what the > current host is capable of only, not whether it is desirable to > actually use them. Just because a host reports that it has q35-2.11.0 > machine type doesn't mean that it should be used. The mgmt app may > only wish to use that if it is available on all hosts in a particular > grouping. The config generation library can't query every host directly > to determine this. The mgmt app may have a way to collate capabilities > info from hosts, but it is probably then stored in a app specific > format and data source, or it may just ends up being a global config > parameter to the mgmt app per host. In other words, you need host capabilities from all hosts as input when generating a new config XML. We already have a format to represent host capabilities defined by libvirt, users of the new system would just need to reproduce the data they got from libvirt and give it to the config generator. Not completely trivial, but maybe worth the effort if you want to benefit from work done by other people to find good defaults? > > There have been a number of times where a feature is available in > libvirt and/or QEMU, and the mgmt app still doesn't yet may still > not wish to use it because it is known broken / incompatible with > certain usage patterns. So the mgmt app would require an arbitrarily > newer libvirt/qemu before considering using it, regardless of > whether host capabilities report it is available. If this happens sometimes, why is it better for the teams maintaining management layers to duplicate the work of finding what works, instead of solving the problem only once? > > > > Even if you have the matching libvirt & QEMU versions, it is not > > > safe to assume the application will want to use the new policy. > > > An application may need live migration compatibility with older > > > versions. Or it may need to retain guaranteed ABI compatibility > > > with the way the VM was previously launched and be using transient > > > guests, generating the XML fresh each time. > > > > Why is that a problem? If you want live migration or ABI > > guarantees, you simply don't use this system to generate a new > > configuration. The same way you don't use the "pc" machine-type > > if you want to ensure compatibility with existing VMs. > > In many mgmt apps, every VM potentially needs live migration, so > unless I'm misunderstanding, you're effectively saying don't ever > use this config generator in these apps. If you only need live migration, you can
Re: [virt-tools-list] [libvirt] Project for profiles and defaults for libvirt domains
On Wed, Mar 21, 2018 at 03:00:41PM -0300, Eduardo Habkost wrote: > On Tue, Mar 20, 2018 at 03:10:12PM +, Daniel P. Berrangé wrote: > > On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote: > > > 1) Default devices/values > > > > > > Libvirt itself must default to whatever values there were before any > > > particular element was introduced due to the fact that it strives to > > > keep the guest ABI stable. That means, for example, that it can't just > > > add -vmcoreinfo option (for KASLR support) or magically add the pvpanic > > > device to all QEMU machines, even though it would be useful, as that > > > would change the guest ABI. > > > > > > For default values this is even more obvious. Let's say someone figures > > > out some "pretty good" default values for various HyperV enlightenment > > > feature tunables. Libvirt can't magically change them, but each one of > > > the projects building on top of it doesn't want to keep that list > > > updated and take care of setting them in every new XML. Some projects > > > don't even expose those to the end user as a knob, while others might. > > > > This gets very tricky, very fast. > > > > Lets say that you have an initial good set of hyperv config > > tunables. Now sometime passes and it is decided that there is a > > different, better set of config tunables. If the module that is > > providing this policy to apps like OpenStack just updates itself > > to provide this new policy, this can cause problems with the > > existing deployed applications in a number of ways. > > > > First the new config probably depends on specific versions of > > libvirt and QEMU, and you can't mandate to consuming apps which > > versions they must be using. [...] > > This is true. > > > [...] So you need a matrix of libvirt + > > QEMU + config option settings. > > But this is not. If config options need support on the lower > levels of the stack (libvirt and/or QEMU and/or KVM and/or host > hardware), it already has to be represented by libvirt host > capabilities somehow, so management layers know it's available. > > This means any new config generation system can (and must) use > host(s) capabilities as input before generating the > configuration. I don't think it is that simple. The capabilities reflect what the current host is capable of only, not whether it is desirable to actually use them. Just because a host reports that it has q35-2.11.0 machine type doesn't mean that it should be used. The mgmt app may only wish to use that if it is available on all hosts in a particular grouping. The config generation library can't query every host directly to determine this. The mgmt app may have a way to collate capabilities info from hosts, but it is probably then stored in a app specific format and data source, or it may just ends up being a global config parameter to the mgmt app per host. There have been a number of times where a feature is available in libvirt and/or QEMU, and the mgmt app still doesn't yet may still not wish to use it because it is known broken / incompatible with certain usage patterns. So the mgmt app would require an arbitrarily newer libvirt/qemu before considering using it, regardless of whether host capabilities report it is available. > > Even if you have the matching libvirt & QEMU versions, it is not > > safe to assume the application will want to use the new policy. > > An application may need live migration compatibility with older > > versions. Or it may need to retain guaranteed ABI compatibility > > with the way the VM was previously launched and be using transient > > guests, generating the XML fresh each time. > > Why is that a problem? If you want live migration or ABI > guarantees, you simply don't use this system to generate a new > configuration. The same way you don't use the "pc" machine-type > if you want to ensure compatibility with existing VMs. In many mgmt apps, every VM potentially needs live migration, so unless I'm misunderstanding, you're effectively saying don't ever use this config generator in these apps. > > The application will have knowledge about when it wants to use new > > vs old hyperv tunable policy, but exposing that to your policy module > > is very tricky because it is inherantly application specific logic > > largely determined by the way the application code is written. > > We have a huge set of features where this is simply not a > problem. For most virtual hardware features, enabling them is > not even a policy decision: it's just about telling the guest > that the feature is now available. QEMU have been enabling new > features in the "pc" machine-type for years. > > Now, why can't higher layers in the stack do something similar? > > The proposal is equivalent to what already happens when people > use the "pc" machine-type in their configurations, but: > 1) the new defaults/features wouldn't be hidden behind a opaque >machine-type name, and
Re: [virt-tools-list] [libvirt] Project for profiles and defaults for libvirt domains
On Tue, Mar 20, 2018 at 03:10:12PM +, Daniel P. Berrangé wrote: > On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote: > > 1) Default devices/values > > > > Libvirt itself must default to whatever values there were before any > > particular element was introduced due to the fact that it strives to > > keep the guest ABI stable. That means, for example, that it can't just > > add -vmcoreinfo option (for KASLR support) or magically add the pvpanic > > device to all QEMU machines, even though it would be useful, as that > > would change the guest ABI. > > > > For default values this is even more obvious. Let's say someone figures > > out some "pretty good" default values for various HyperV enlightenment > > feature tunables. Libvirt can't magically change them, but each one of > > the projects building on top of it doesn't want to keep that list > > updated and take care of setting them in every new XML. Some projects > > don't even expose those to the end user as a knob, while others might. > > This gets very tricky, very fast. > > Lets say that you have an initial good set of hyperv config > tunables. Now sometime passes and it is decided that there is a > different, better set of config tunables. If the module that is > providing this policy to apps like OpenStack just updates itself > to provide this new policy, this can cause problems with the > existing deployed applications in a number of ways. > > First the new config probably depends on specific versions of > libvirt and QEMU, and you can't mandate to consuming apps which > versions they must be using. [...] This is true. > [...] So you need a matrix of libvirt + > QEMU + config option settings. But this is not. If config options need support on the lower levels of the stack (libvirt and/or QEMU and/or KVM and/or host hardware), it already has to be represented by libvirt host capabilities somehow, so management layers know it's available. This means any new config generation system can (and must) use host(s) capabilities as input before generating the configuration. > > Even if you have the matching libvirt & QEMU versions, it is not > safe to assume the application will want to use the new policy. > An application may need live migration compatibility with older > versions. Or it may need to retain guaranteed ABI compatibility > with the way the VM was previously launched and be using transient > guests, generating the XML fresh each time. Why is that a problem? If you want live migration or ABI guarantees, you simply don't use this system to generate a new configuration. The same way you don't use the "pc" machine-type if you want to ensure compatibility with existing VMs. > > The application will have knowledge about when it wants to use new > vs old hyperv tunable policy, but exposing that to your policy module > is very tricky because it is inherantly application specific logic > largely determined by the way the application code is written. We have a huge set of features where this is simply not a problem. For most virtual hardware features, enabling them is not even a policy decision: it's just about telling the guest that the feature is now available. QEMU have been enabling new features in the "pc" machine-type for years. Now, why can't higher layers in the stack do something similar? The proposal is equivalent to what already happens when people use the "pc" machine-type in their configurations, but: 1) the new defaults/features wouldn't be hidden behind a opaque machine-type name, and would appear in the domain XML explicitly; 2) the higher layers won't depend on QEMU introducing a new machine-type just to have new features enabled by default; 3) features that depend on host capabilities but are available on all hosts in a cluster can now be enabled automatically if desired (which is something QEMU can't do because it doesn't have enough information about the other hosts). Choosing reasonable defaults might not be a trivial problem, but the current approach of pushing the responsibility to management layers doesn't improve the situation. [...] > > 2) Policies [...] > > 3) Abstracting the XML [...] > > 4) Identifying devices properly [...] > > 5) Generating the right XML snippet for device hot-(un)plug [...] These parts are trickier and I need to read the discussion more carefully before replying. -- Eduardo ___ virt-tools-list mailing list virt-tools-list@redhat.com https://www.redhat.com/mailman/listinfo/virt-tools-list
