Re: [libvirt] [kubevirt-dev] Re: [virt-tools-list] Project for profiles and defaults for libvirt domains

[Daniel P . Berrangé]

On Thu, Mar 22, 2018 at 03:54:01PM +0100, Martin Kletzander wrote:
> > 
> > > One more thing could be automatically figuring out best values based on
> > > libosinfo-provided data.
> > > 
> > > 2) Policies
> > > 
> > > Lot of the time there are parts of the domain definition that need to be
> > > added, but nobody really cares about them.  Sometimes it's enough to
> > > have few templates, another time you might want to have a policy
> > > per-scenario and want to combine them in various ways.  For example with
> > > the data provided by point 1).
> > > 
> > > For example if you want PCI-Express, you need the q35 machine type, but
> > > you don't really want to care about the machine type.  Or you want to
> > > use SPICE, but you don't want to care about adding QXL.
> > > 
> > > What if some of these policies could be specified once (using some DSL
> > > for example), and used by virtuned to merge them in a unified and
> > > predictable way?
> > > 
> > > 3) Abstracting the XML
> > > 
> > > This is probably just usable for stateless apps, but it might happen
> > > that some apps don't really want to care about the XML at all.  They
> > > just want an abstract view of the domain, possibly add/remove a device
> > > and that's it.  We could do that as well.  I can't really tell how much
> > > of a demand there is for it, though.
> > 
> > It is safe to say that applications do not want to touch XML at all.
> > Any non-trivial application has created an abstraction around XML,
> > so that they have an API to express what they want, rather than
> > manipulating of strings to format/parse XML.
> > 
> 
> Sure, this was just meant to be a question as to whether it's worth
> pursuing or not.  You make a good point on why it is not (at least for
> existing apps).
> 
> However, since this was optional, the way this would look without the
> XML abstraction is that both input and output would be valid domain
> definitions, ultimately resulting in something similar to virt-xml with
> the added benefit of applying a policy from a file/string either
> supplied by the application itself.  Whether that policy was taken from
> a common repository of such knowledge is orthogonal to this idea.  Since
> you would work with the same data, the upgrade could be incremental as
> you'd only let virtuned fill in values for new options and could slowly
> move on to using it for some pre-existing ones.  None of the previous
> approaches did this, if I'm not mistaken.  Of course it gets more
> difficult when you need to expose all the bits libvirt does and keep
> them in sync (as you write below).

That has implications for how mgmt app deals with XML. Nova has object
models for representing XML in memory, but it doesn't aim to have
loss-less roundtrip from parse -> object -> format. So if Nova gets
basic XML from virttuned, parses it into its object to let it set
more fields and then formats it again, chances are it will have lost
a bunch of stuff from virttuned. Of course if you know about this
need upfront you can design the application such that it can safely
round-trip, but this is just example of problem with integrating to
existing apps.

The other thing that concerns is that there are dependancies between
different bits of XML for a given device. ie if feature X is set to
a certain value, that prevents use of feature Y. So if virttuned
sets feature X, but  the downstream application uses feature Y, the
final result can be incompatible. The application won't know this
because it doesn't control what stuff virttuned would be setting.
This can in turn cause ordering constraints.

eg the application needs to say that virtio-net is being used, then
virttuned can set some defaults like enabling vhost-net, and then
the application can fill in more bits that it cares about. Or if
we let virttuned go first, setting virtio-net model + vhost-net,
then application wants to change model to e1000e, it has to be
aware that it must now delete the vhost-net bit that virtuned
added. This ends up being more complicated that just ignoring
virttuned and coding up use of vhost-net in application code.


> > This is the same kind of problem we faced wrt libvirt-gconfig and
> > libvirt-gobject usage from virt-manager - it has an extensive code
> > base that already works, and rewriting it to use something new
> > is alot of work for no short-term benefit. libvirt-gconfig/gobject
> > were supposed to be the "easy" bits for virt-manager to adopt, as
> > they don't really include much logic that would step on virt-manager's
> > toes. libvirt-designer was going to be a very opinionated library
> > and in retrospective that makes it even harder to consider adopting
> > it for usage in virt-manager, as it'll have signficant liklihood
> > of making functionally significant changes in behaviour.
> > 
> 
> The initial idea (which I forgot to mention) was that all the decisions
> libvirt currently does (so that it keeps the guest ABI stable) would be
> moved into 

Re: [libvirt] [kubevirt-dev] Re: [virt-tools-list] Project for profiles and defaults for libvirt domains

[Eduardo Habkost]

On Thu, Mar 22, 2018 at 09:56:12AM +, Daniel P. Berrangé wrote:
> On Wed, Mar 21, 2018 at 04:34:23PM -0300, Eduardo Habkost wrote:
> > On Wed, Mar 21, 2018 at 06:39:52PM +, Daniel P. Berrangé wrote:
> > > On Wed, Mar 21, 2018 at 03:00:41PM -0300, Eduardo Habkost wrote:
> > > > On Tue, Mar 20, 2018 at 03:10:12PM +, Daniel P. Berrangé wrote:
> > > > > On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
> > > > > > 1) Default devices/values
> > > > > > 
> > > > > > Libvirt itself must default to whatever values there were before any
> > > > > > particular element was introduced due to the fact that it strives to
> > > > > > keep the guest ABI stable.  That means, for example, that it can't 
> > > > > > just
> > > > > > add -vmcoreinfo option (for KASLR support) or magically add the 
> > > > > > pvpanic
> > > > > > device to all QEMU machines, even though it would be useful, as that
> > > > > > would change the guest ABI.
> > > > > > 
> > > > > > For default values this is even more obvious.  Let's say someone 
> > > > > > figures
> > > > > > out some "pretty good" default values for various HyperV 
> > > > > > enlightenment
> > > > > > feature tunables.  Libvirt can't magically change them, but each 
> > > > > > one of
> > > > > > the projects building on top of it doesn't want to keep that list
> > > > > > updated and take care of setting them in every new XML.  Some 
> > > > > > projects
> > > > > > don't even expose those to the end user as a knob, while others 
> > > > > > might.
> > > > > 
> > > > > This gets very tricky, very fast.
> > > > > 
> > > > > Lets say that you have an initial good set of hyperv config
> > > > > tunables. Now sometime passes and it is decided that there is a
> > > > > different, better set of config tunables. If the module that is
> > > > > providing this policy to apps like OpenStack just updates itself
> > > > > to provide this new policy, this can cause problems with the
> > > > > existing deployed applications in a number of ways.
> > > > > 
> > > > > First the new config probably depends on specific versions of
> > > > > libvirt and QEMU,  and you can't mandate to consuming apps which
> > > > > versions they must be using.  [...]
> > > > 
> > > > This is true.
> > > > 
> > > > >   [...]  So you need a matrix of libvirt +
> > > > > QEMU + config option settings.
> > > > 
> > > > But this is not.  If config options need support on the lower
> > > > levels of the stack (libvirt and/or QEMU and/or KVM and/or host
> > > > hardware), it already has to be represented by libvirt host
> > > > capabilities somehow, so management layers know it's available.
> > > > 
> > > > This means any new config generation system can (and must) use
> > > > host(s) capabilities as input before generating the
> > > > configuration.
> > > 
> > > I don't think it is that simple. The capabilities reflect what the
> > > current host is capable of only, not whether it is desirable to
> > > actually use them. Just because a host reports that it has q35-2.11.0
> > > machine type doesn't mean that it should be used. The mgmt app may
> > > only wish to use that if it is available on all hosts in a particular
> > > grouping. The config generation library can't query every host directly
> > > to determine this. The mgmt app may have a way to collate capabilities
> > > info from hosts, but it is probably then stored in a app specific
> > > format and data source, or it may just ends up being a global config
> > > parameter to the mgmt app per host.
> > 
> > In other words, you need host capabilities from all hosts as
> > input when generating a new config XML.  We already have a format
> > to represent host capabilities defined by libvirt, users of the
> > new system would just need to reproduce the data they got from
> > libvirt and give it to the config generator.
> 
> Things aren't that simple - when openstack reports info from each host
> it doesn't do it in any libvirt format - it uses an arbitrary format it
> defines itself. Going from libvirt host capabilities to the app specific
> format and back to libvirt host capabilities will loose information.
> Then you also have matter of hosts coming & going over time, so fragile
> to assume that the set of host capabilities you currently see are
> representative of the steady state you desire.

Well, then the management layer should stop losing useful data.  ;)

(But I understand that this is not that simple)

> 
> > Not completely trivial, but maybe worth the effort if you want to
> > benefit from work done by other people to find good defaults?
> 
> Perhaps, but there's many ways to share the work of figuring out
> good defaults. Beyond what's represented in libosinfo database,
> no one has even tried to document what current desirable defaults
> are. Jumping straight from no documented best practice, to lets
> build a API is a big ask, particularly when the suggestion involves
> major architectural changes to 

Re: [libvirt] [kubevirt-dev] Re: [virt-tools-list] Project for profiles and defaults for libvirt domains

[Michal Skrivanek]

> On 22 Mar 2018, at 10:37, Daniel P. Berrangé  wrote:
> 
> On Thu, Mar 22, 2018 at 09:40:40AM +0100, Pavel Hrdina wrote:
>> On Tue, Mar 20, 2018 at 03:10:12PM +, Daniel P. Berrangé wrote:
 - I understand OpenStack has some really sensible and wisely chosen
 and/or tested default values.
>>> 
>>> In terms of default devices and OS specific choices, OpenStack's
>>> decisions have been largely inspired by previous work in oVirt
>>> and / or virt-manager. So there's obviously overlap in the
>>> conceptual area, but there's also plenty that is very specific
>>> to OpenStack - untangling the two extract the common bits from
>>> the app specific bits is hard. 
>> 
>> This would be handled by the application specific policies.  The
>> virtuned will have some reasonable defaults that are known to work in
>> most cases and suits the majority of users, but it's clear that
>> sometimes you need some specific defaults and you would provide them
>> via the application policy.
>> 
>> For example, to create a XML for windows guest the virtuned would not
>> probably select virtio devices because there are no drivers for them
>> in the standard windows installation, however, some management
>> application may have customized preinstalled disk images or customized
>> ISO images or it may be able to provide the drivers any other way, so
>> they would specify in the application policy that for windows guest
>> virtuned should use virtio as a default device model.
> 
> As soon as we talk about configuring hardware specific to a guest
> OS, then that is in scope of existing libosinfo project, not something
> we should create a new project for.
> 
> 
>> This is probably the hardest part of creating higher level API on top
>> of libvirt, not every project may be willing to rewrite their existing
>> code.  On the other hand, I know that for example Cockpit would benefit
>> from the virtuned providing this functionality via REST API.
>> 
>> It's a chicken and egg problem, but if we can gather input from all the
>> existing projects that have their own implementation and figure out how
>> to make virtuned usable for all of them they might consider to start
>> using it.
> 
> I don't doubt that Cockpit would like like, but based on previous efforts
> we've made I'm sceptical that anything beyond Cockpit would use any new
> API. 

We have an opportunity to make kubevirt right, too. Sure, the project is 
halfway through with own presets and custom code (just repeating oVirt and 
Openstack), but it’s not too late to change things there, and a remote API fits 
well

> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o-https://fstop138.berrange.com :|
> |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "kubevirt-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to kubevirt-dev+unsubscr...@googlegroups.com.
> To post to this group, send email to kubevirt-...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/kubevirt-dev/20180322093722.GB3583%40redhat.com.
> For more options, visit https://groups.google.com/d/optout.


--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [kubevirt-dev] Re: [virt-tools-list] Project for profiles and defaults for libvirt domains

[Daniel P . Berrangé]

On Wed, Mar 21, 2018 at 04:34:23PM -0300, Eduardo Habkost wrote:
> On Wed, Mar 21, 2018 at 06:39:52PM +, Daniel P. Berrangé wrote:
> > On Wed, Mar 21, 2018 at 03:00:41PM -0300, Eduardo Habkost wrote:
> > > On Tue, Mar 20, 2018 at 03:10:12PM +, Daniel P. Berrangé wrote:
> > > > On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
> > > > > 1) Default devices/values
> > > > > 
> > > > > Libvirt itself must default to whatever values there were before any
> > > > > particular element was introduced due to the fact that it strives to
> > > > > keep the guest ABI stable.  That means, for example, that it can't 
> > > > > just
> > > > > add -vmcoreinfo option (for KASLR support) or magically add the 
> > > > > pvpanic
> > > > > device to all QEMU machines, even though it would be useful, as that
> > > > > would change the guest ABI.
> > > > > 
> > > > > For default values this is even more obvious.  Let's say someone 
> > > > > figures
> > > > > out some "pretty good" default values for various HyperV enlightenment
> > > > > feature tunables.  Libvirt can't magically change them, but each one 
> > > > > of
> > > > > the projects building on top of it doesn't want to keep that list
> > > > > updated and take care of setting them in every new XML.  Some projects
> > > > > don't even expose those to the end user as a knob, while others might.
> > > > 
> > > > This gets very tricky, very fast.
> > > > 
> > > > Lets say that you have an initial good set of hyperv config
> > > > tunables. Now sometime passes and it is decided that there is a
> > > > different, better set of config tunables. If the module that is
> > > > providing this policy to apps like OpenStack just updates itself
> > > > to provide this new policy, this can cause problems with the
> > > > existing deployed applications in a number of ways.
> > > > 
> > > > First the new config probably depends on specific versions of
> > > > libvirt and QEMU,  and you can't mandate to consuming apps which
> > > > versions they must be using.  [...]
> > > 
> > > This is true.
> > > 
> > > >   [...]  So you need a matrix of libvirt +
> > > > QEMU + config option settings.
> > > 
> > > But this is not.  If config options need support on the lower
> > > levels of the stack (libvirt and/or QEMU and/or KVM and/or host
> > > hardware), it already has to be represented by libvirt host
> > > capabilities somehow, so management layers know it's available.
> > > 
> > > This means any new config generation system can (and must) use
> > > host(s) capabilities as input before generating the
> > > configuration.
> > 
> > I don't think it is that simple. The capabilities reflect what the
> > current host is capable of only, not whether it is desirable to
> > actually use them. Just because a host reports that it has q35-2.11.0
> > machine type doesn't mean that it should be used. The mgmt app may
> > only wish to use that if it is available on all hosts in a particular
> > grouping. The config generation library can't query every host directly
> > to determine this. The mgmt app may have a way to collate capabilities
> > info from hosts, but it is probably then stored in a app specific
> > format and data source, or it may just ends up being a global config
> > parameter to the mgmt app per host.
> 
> In other words, you need host capabilities from all hosts as
> input when generating a new config XML.  We already have a format
> to represent host capabilities defined by libvirt, users of the
> new system would just need to reproduce the data they got from
> libvirt and give it to the config generator.

Things aren't that simple - when openstack reports info from each host
it doesn't do it in any libvirt format - it uses an arbitrary format it
defines itself. Going from libvirt host capabilities to the app specific
format and back to libvirt host capabilities will loose information.
Then you also have matter of hosts coming & going over time, so fragile
to assume that the set of host capabilities you currently see are
representative of the steady state you desire.

> Not completely trivial, but maybe worth the effort if you want to
> benefit from work done by other people to find good defaults?

Perhaps, but there's many ways to share the work of figuring out
good defaults. Beyond what's represented in libosinfo database,
no one has even tried to document what current desirable defaults
are. Jumping straight from no documented best practice, to lets
build a API is a big ask, particularly when the suggestion involves
major architectural changes to any app that wants to use it.

For most immediate benefit actually documenting some best practice
would be the most tangible win for application developers, as they
can much more easily adapt existing code to follow it. ALso expanding
range of info we record in libosinfo would be beneficial, since there
is still plenty of OS specific data not captured. Not to mention that
most applications