Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Matt Riedemann]

On 9/30/2016 11:03 AM, Matthew Booth wrote:



I thought it was a counter-argument (unless I've misunderstood). If you
migrate the instance as-is without modification, you don't need to worry
about whether it's currently a rescue instance. This problem goes away.

The major complication I can think of is things which genuinely must
change during a migration, for example cpu pinning.

Matt

--
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



I'm glad this already came up in this thread as I was just reviewing the 
re-proposal for this spec:


https://review.openstack.org/#/c/347161/

There are other issues in the spec which is why I have a -1 on there, 
but I thought it was going directly against what Matt Booth is talking 
about doing wrt storing the guest xml so that we don't need to rely on 
the nova database when recreating a guest, like on reboot - which is 
exactly what that spec is proposing, just hard-reboot the live migrated 
guest on unrescue.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Daniel P. Berrange]

On Fri, Sep 30, 2016 at 04:36:31PM +, Murray, Paul (HP Cloud) wrote:
> 
> We have a problem migrating rescued instances that has a fix in progress based
> on regenerating the xml on unrescue, see:
> 
>  https://blueprints.launchpad.net/nova/+spec/live-migrate-rescued-instances
> 
> That might be another case for generating the xml.
> 
> I thought it was a counter-argument (unless I've misunderstood). If you 
> migrate the instance as-is without modification, you don't need to worry 
> about whether it's currently a rescue instance. This problem goes away.
> 
> The major complication I can think of is things which genuinely must change 
> during a migration, for example cpu pinning.
> 
> Rescue currently saves the libvirt xml in a separate file and replaces it
> with new xml to define a vm with a rescue boot disk; to unrescue it replaces
> the libvirt xml used for the rescue with the saved file to go back to the
> original state. On migration that saved xml would be lost because its an
> arbitrary file that is not handled in the migration. The spec above relies
> on the fact that we do not need to save it or copy it because we can recreate
> it from nova. So yes, the migration just works for the rescued vm, but when
> it is unrescued the original libvirt xml would be regenerated.

During rescue, nova should really not be touching the XML on disk at all. That
should have been left to reflect the "normal" XML of the guest. Instead nova
should have just called 'createXML' method, to boot the guest with a one-time
different XML config. There is no reason to define the XM on disk with the
custom rescue config.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Daniel P. Berrange]

On Mon, Oct 03, 2016 at 10:11:34AM +0300, Timofei Durakov wrote:
> Hi team,
> 
> I agree that it's kind of strange thing that nova dumps xml definition to
> the disk but doesn't use it(at least I do not aware of it).
> How the proposed changed would be aligned with other drivers? The worst
> case I could imagine here is that libvirt has an xml as a source of truth,
> while others use nova for the same purposes. Taking that into account, the
> question here would be:  why not to store all required information(e.g.
> boot order) in DB instead?

That is duplicating information already stored in libvirt - any time you
change the guest you have the job of updating the DB copy to mirror this
change. This gets particularly fun (aka error prone) during migration
if there's a failure during migration, as you can get the two copies out
of sync (eg if libvirt completes, but something causes an exception in
nova's post-migration logic).

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Roman Podoliaka]

Timofei,

On Mon, Oct 3, 2016 at 10:11 AM, Timofei Durakov  wrote:
> Hi team,
> Taking that into account, the
> question here would be:  why not to store all required information(e.g. boot
> order) in DB instead?

I think, we definitely could do that, just like we currently preserve
the order of NICs on reboot. I'm not sure it fully resolves Matt's
concerns, though, when it comes to preserving PCI device addresses. If
we were to keep the Nova DB as our source of truth, then we'd probably
need the libvirt driver to generate these addresses on creation of a
domain in a consistent fashion (assuming we store the order of devices
in the DB).

Thanks,
Roman

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Timofei Durakov]

Hi team,

I agree that it's kind of strange thing that nova dumps xml definition to
the disk but doesn't use it(at least I do not aware of it).
How the proposed changed would be aligned with other drivers? The worst
case I could imagine here is that libvirt has an xml as a source of truth,
while others use nova for the same purposes. Taking that into account, the
question here would be:  why not to store all required information(e.g.
boot order) in DB instead?

Timofey


On Fri, Sep 30, 2016 at 7:36 PM, Murray, Paul (HP Cloud) <pmur...@hpe.com>
wrote:

>
>
> From: Matthew Booth
> Reply-To: "openstack-dev@lists.openstack.org"
> Date: Friday, 30 September 2016 at 17:03
> To: "openstack-dev@lists.openstack.org"
> Subject: Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain
> XML canonical
>
> On Fri, Sep 30, 2016 at 4:38 PM, Murray, Paul (HP Cloud) <pmur...@hpe.com>
> wrote:
>
>>
>>
>>
>>
>>
>> On 27/09/2016, 18:12, "Daniel P. Berrange" <berra...@redhat.com> wrote:
>>
>> >On Tue, Sep 27, 2016 at 10:40:34AM -0600, Chris Friesen wrote:
>> >> On 09/27/2016 10:17 AM, Matthew Booth wrote:
>> >>
>> >> > I think we should be able to create a domain, but once created we
>> should never
>> >> > redefine a domain. We can do adding and removing devices dynamically
>> using
>> >> > libvirt's apis, secure in the knowledge that libvirt will persist
>> this for us.
>> >> > When we upgrade the host, libvirt can ensure we don't break guests
>> which are on
>> >> > it. Evacuate should be pretty much the only reason to start again.
>> >>
>> >> Sounds interesting.  How would you handle live migration?
>> >>
>> >> Currently we regenerate the XML file on the destination from the nova
>> DB.  I
>> >> guess in your proposal we'd need some way of copying the XML file from
>> the
>> >> source to the dest, and then modifying the appropriate segments to
>> adjust
>> >> things like CPU/NUMA pinning?
>> >
>> >Use the flag VIR_MIGRATE_PERSIST_XML and libvirt will write out the
>> >new persistent XML on the target host automatically.
>>
>> We have a problem migrating rescued instances that has a fix in progress
>> based
>> on regenerating the xml on unrescue, see:
>>
>>  https://blueprints.launchpad.net/nova/+spec/live-migrate-re
>> scued-instances
>>
>> That might be another case for generating the xml.
>>
>
> I thought it was a counter-argument (unless I've misunderstood). If you
> migrate the instance as-is without modification, you don't need to worry
> about whether it's currently a rescue instance. This problem goes away.
>
> The major complication I can think of is things which genuinely must
> change during a migration, for example cpu pinning.
>
>
> Rescue currently saves the libvirt xml in a separate file and replaces it
> with new xml to define a vm with a rescue boot disk; to unrescue it
> replaces the libvirt xml used for the rescue with the saved file to go back
> to the original state. On migration that saved xml would be lost because
> its an arbitrary file that is not handled in the migration. The spec above
> relies on the fact that we do not need to save it or copy it because we can
> recreate it from nova. So yes, the migration just works for the rescued vm,
> but when it is unrescued the original libvirt xml would be regenerated.
>
> Paul
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Murray, Paul (HP Cloud)]

From: Matthew Booth
Reply-To: 
"openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>"
Date: Friday, 30 September 2016 at 17:03
To: 
"openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>"
Subject: Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML 
canonical

On Fri, Sep 30, 2016 at 4:38 PM, Murray, Paul (HP Cloud) 
<pmur...@hpe.com<mailto:pmur...@hpe.com>> wrote:





On 27/09/2016, 18:12, "Daniel P. Berrange" 
<berra...@redhat.com<mailto:berra...@redhat.com>> wrote:

>On Tue, Sep 27, 2016 at 10:40:34AM -0600, Chris Friesen wrote:
>> On 09/27/2016 10:17 AM, Matthew Booth wrote:
>>
>> > I think we should be able to create a domain, but once created we should 
>> > never
>> > redefine a domain. We can do adding and removing devices dynamically using
>> > libvirt's apis, secure in the knowledge that libvirt will persist this for 
>> > us.
>> > When we upgrade the host, libvirt can ensure we don't break guests which 
>> > are on
>> > it. Evacuate should be pretty much the only reason to start again.
>>
>> Sounds interesting.  How would you handle live migration?
>>
>> Currently we regenerate the XML file on the destination from the nova DB.  I
>> guess in your proposal we'd need some way of copying the XML file from the
>> source to the dest, and then modifying the appropriate segments to adjust
>> things like CPU/NUMA pinning?
>
>Use the flag VIR_MIGRATE_PERSIST_XML and libvirt will write out the
>new persistent XML on the target host automatically.

We have a problem migrating rescued instances that has a fix in progress based
on regenerating the xml on unrescue, see:

 https://blueprints.launchpad.net/nova/+spec/live-migrate-rescued-instances

That might be another case for generating the xml.

I thought it was a counter-argument (unless I've misunderstood). If you migrate 
the instance as-is without modification, you don't need to worry about whether 
it's currently a rescue instance. This problem goes away.

The major complication I can think of is things which genuinely must change 
during a migration, for example cpu pinning.

Rescue currently saves the libvirt xml in a separate file and replaces it with 
new xml to define a vm with a rescue boot disk; to unrescue it replaces the 
libvirt xml used for the rescue with the saved file to go back to the original 
state. On migration that saved xml would be lost because its an arbitrary file 
that is not handled in the migration. The spec above relies on the fact that we 
do not need to save it or copy it because we can recreate it from nova. So yes, 
the migration just works for the rescued vm, but when it is unrescued the 
original libvirt xml would be regenerated.

Paul

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Matthew Booth]

On Fri, Sep 30, 2016 at 4:38 PM, Murray, Paul (HP Cloud) 
wrote:

>
>
>
>
>
> On 27/09/2016, 18:12, "Daniel P. Berrange"  wrote:
>
> >On Tue, Sep 27, 2016 at 10:40:34AM -0600, Chris Friesen wrote:
> >> On 09/27/2016 10:17 AM, Matthew Booth wrote:
> >>
> >> > I think we should be able to create a domain, but once created we
> should never
> >> > redefine a domain. We can do adding and removing devices dynamically
> using
> >> > libvirt's apis, secure in the knowledge that libvirt will persist
> this for us.
> >> > When we upgrade the host, libvirt can ensure we don't break guests
> which are on
> >> > it. Evacuate should be pretty much the only reason to start again.
> >>
> >> Sounds interesting.  How would you handle live migration?
> >>
> >> Currently we regenerate the XML file on the destination from the nova
> DB.  I
> >> guess in your proposal we'd need some way of copying the XML file from
> the
> >> source to the dest, and then modifying the appropriate segments to
> adjust
> >> things like CPU/NUMA pinning?
> >
> >Use the flag VIR_MIGRATE_PERSIST_XML and libvirt will write out the
> >new persistent XML on the target host automatically.
>
> We have a problem migrating rescued instances that has a fix in progress
> based
> on regenerating the xml on unrescue, see:
>
>  https://blueprints.launchpad.net/nova/+spec/live-migrate-
> rescued-instances
>
> That might be another case for generating the xml.
>

I thought it was a counter-argument (unless I've misunderstood). If you
migrate the instance as-is without modification, you don't need to worry
about whether it's currently a rescue instance. This problem goes away.

The major complication I can think of is things which genuinely must change
during a migration, for example cpu pinning.

Matt

-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Murray, Paul (HP Cloud)]

On 27/09/2016, 18:12, "Daniel P. Berrange"  wrote:

>On Tue, Sep 27, 2016 at 10:40:34AM -0600, Chris Friesen wrote:
>> On 09/27/2016 10:17 AM, Matthew Booth wrote:
>> 
>> > I think we should be able to create a domain, but once created we should 
>> > never
>> > redefine a domain. We can do adding and removing devices dynamically using
>> > libvirt's apis, secure in the knowledge that libvirt will persist this for 
>> > us.
>> > When we upgrade the host, libvirt can ensure we don't break guests which 
>> > are on
>> > it. Evacuate should be pretty much the only reason to start again.
>> 
>> Sounds interesting.  How would you handle live migration?
>> 
>> Currently we regenerate the XML file on the destination from the nova DB.  I
>> guess in your proposal we'd need some way of copying the XML file from the
>> source to the dest, and then modifying the appropriate segments to adjust
>> things like CPU/NUMA pinning?
>
>Use the flag VIR_MIGRATE_PERSIST_XML and libvirt will write out the
>new persistent XML on the target host automatically.

We have a problem migrating rescued instances that has a fix in progress based
on regenerating the xml on unrescue, see:

 https://blueprints.launchpad.net/nova/+spec/live-migrate-rescued-instances 

That might be another case for generating the xml.

Paul


>
>Regards,
>Daniel
>-- 
>|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
>|: http://libvirt.org  -o- http://virt-manager.org :|
>|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Artom Lifshitz]

By coincidence I've just written up a spec [1] that proposes an
admittedly very generic mechanism to solve this problem. I was coming
at it from the perspective of keeping the relative order of PCI device
addresses constant across evacuations. In that spec, I propose letting
the virt driver store blobs of data in the database (with some
caveats, obviously). This can then be used, for example, by the
libvirt driver to persist instance XML throughout an instance's
lifetime.

I agree with Dan Berranage that it's an overkill solution - we can
simply use libvirt itself as the storage mechanism and not have this
duplicate blob floating in the database confusing us about which
source of truth do we really want to use - except for the evacuation
edge case. When evacuating, the source host is unavailable, thus we're
unable to retrieve the instance XML from it. Also, in my conversation
with Claudiu (and hopefully he can chime in here and confirm I'm not
putting words in his mouth), the Hyper-V folks are potentially
interested in something like the driver private storage mechanism
proposed in the spec [1] to use with their new VM config export/import
feature [2].

[1] https://review.openstack.org/#/c/377806/
[2] https://review.openstack.org/#/c/340908/

On Tue, Sep 27, 2016 at 12:31 PM, Daniel P. Berrange
 wrote:
> On Tue, Sep 27, 2016 at 05:17:29PM +0100, Matthew Booth wrote:
>> Currently the libvirt driver (mostly) considers the nova db canonical. That
>> is, we can throw away libvirt's domain XML at any time and recreate it from
>> Nova. Anywhere that doesn't assume this is a bug, because whatever
>> direction we choose we don't need 2 different sources of truth. The
>> thinking behind this is that we should always know what we told libvirt,
>> and if we lose that information then that's a bug.
>>
>> This is true to a degree, and it's the reason I proposed the persistent
>> instance storage metadata spec: we lose track of how we configured an
>> instance's storage. I realised recently that this isn't the whole story,
>> though. Libvirt also automatically creates a bunch of state for us which we
>> didn't specify explicitly. We lose this every time we drop it and recreate.
>> For example, consider device addressing and ordering:
>>
>> $ nova boot ...
>>
>> We tell libvirt to give us a root disk, config disk, and a memballoon
>> device (amongst other things).
>>
>> Libvirt assigns pci addresses to all of these things.
>>
>> $ nova volume-attach ...
>>
>> We tell libvirt to create a new disk attached to the given volume.
>>
>> Libvirt assigns it a pci address.
>>
>> $ nova reboot
>>
>> We throw away libvirt's domain xml and create a new one from scratch.
>>
>> Libvirt assigns new addresses for all of these devices.
>>
>> Before reboot, the device order was: root disk, config disk, memballoon,
>> volume. After reboot the device order is: root disk, volume, config disk,
>> memballoon. Not only have all our devices changed address, which makes
>> Windows sad and paranoid about its licensing, and causes it to offline
>> volumes under certain circumstances, but our disks have been reordered.
>
> It is worth pointing out that we do have the device metadata role
> tagging support now, which lets guest OS identify devices automatically
> at startup. In theory you could say guests should rely on using that
> on *every* boot, not merely the first boot after provisioning.
>
> I think there is reasonable case to be made, however, that we should
> maintain a stable device configuration for an instance after its
> initial boot attempt. Arbitrarily changing hardware config on every
> reboot is being gratuitously nasty to guest admins. The example about
> causing Windows to require license reaactivation is on its own, enough
> of a reason to ensure stable hardware once initial provisioning is
> done.
>
>
>> This isn't all we've thrown away, though. Libvirt also gave us a default
>> machine type. When we create a new domain we'll get a new default machine
>> type. If libvirt has been upgraded, eg during host maintenance, this isn't
>> necessarily what it was before. Again, this can make our guests sad. Same
>> goes for CPU model, default devices, and probably many more things I
>> haven't thought of.
>
> Yes indeed.
>
>> Also... we lost the storage configuration of the guest: the information I
>> propose to persist in persistent instance storage metadata.
>>
>> We could store all of this information in Nova, but with the possible
>> exception of storage metadata it really isn't at the level of 'management':
>> it's the minutia of the hypervisor. In order to persist all of these things
>> in Nova we'd have to implement them explicitly, and when libvirt/kvm grows
>> more stuff we'll have to do that too. We'll need to mirror the
>> functionality of libvirt in Nova, feature for feature. This is a red flag
>> for me, and I think it means we should switch to libvirt being canonical.
>>
>> I think we should be able to 

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Daniel P. Berrange]

On Tue, Sep 27, 2016 at 10:40:34AM -0600, Chris Friesen wrote:
> On 09/27/2016 10:17 AM, Matthew Booth wrote:
> 
> > I think we should be able to create a domain, but once created we should 
> > never
> > redefine a domain. We can do adding and removing devices dynamically using
> > libvirt's apis, secure in the knowledge that libvirt will persist this for 
> > us.
> > When we upgrade the host, libvirt can ensure we don't break guests which 
> > are on
> > it. Evacuate should be pretty much the only reason to start again.
> 
> Sounds interesting.  How would you handle live migration?
> 
> Currently we regenerate the XML file on the destination from the nova DB.  I
> guess in your proposal we'd need some way of copying the XML file from the
> source to the dest, and then modifying the appropriate segments to adjust
> things like CPU/NUMA pinning?

Use the flag VIR_MIGRATE_PERSIST_XML and libvirt will write out the
new persistent XML on the target host automatically.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Chris Friesen]

On 09/27/2016 10:17 AM, Matthew Booth wrote:


I think we should be able to create a domain, but once created we should never
redefine a domain. We can do adding and removing devices dynamically using
libvirt's apis, secure in the knowledge that libvirt will persist this for us.
When we upgrade the host, libvirt can ensure we don't break guests which are on
it. Evacuate should be pretty much the only reason to start again.


Sounds interesting.  How would you handle live migration?

Currently we regenerate the XML file on the destination from the nova DB.  I 
guess in your proposal we'd need some way of copying the XML file from the 
source to the dest, and then modifying the appropriate segments to adjust things 
like CPU/NUMA pinning?


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Daniel P. Berrange]

On Tue, Sep 27, 2016 at 05:17:29PM +0100, Matthew Booth wrote:
> Currently the libvirt driver (mostly) considers the nova db canonical. That
> is, we can throw away libvirt's domain XML at any time and recreate it from
> Nova. Anywhere that doesn't assume this is a bug, because whatever
> direction we choose we don't need 2 different sources of truth. The
> thinking behind this is that we should always know what we told libvirt,
> and if we lose that information then that's a bug.
> 
> This is true to a degree, and it's the reason I proposed the persistent
> instance storage metadata spec: we lose track of how we configured an
> instance's storage. I realised recently that this isn't the whole story,
> though. Libvirt also automatically creates a bunch of state for us which we
> didn't specify explicitly. We lose this every time we drop it and recreate.
> For example, consider device addressing and ordering:
> 
> $ nova boot ...
> 
> We tell libvirt to give us a root disk, config disk, and a memballoon
> device (amongst other things).
> 
> Libvirt assigns pci addresses to all of these things.
> 
> $ nova volume-attach ...
> 
> We tell libvirt to create a new disk attached to the given volume.
> 
> Libvirt assigns it a pci address.
> 
> $ nova reboot
> 
> We throw away libvirt's domain xml and create a new one from scratch.
> 
> Libvirt assigns new addresses for all of these devices.
> 
> Before reboot, the device order was: root disk, config disk, memballoon,
> volume. After reboot the device order is: root disk, volume, config disk,
> memballoon. Not only have all our devices changed address, which makes
> Windows sad and paranoid about its licensing, and causes it to offline
> volumes under certain circumstances, but our disks have been reordered.

It is worth pointing out that we do have the device metadata role
tagging support now, which lets guest OS identify devices automatically
at startup. In theory you could say guests should rely on using that
on *every* boot, not merely the first boot after provisioning.

I think there is reasonable case to be made, however, that we should
maintain a stable device configuration for an instance after its
initial boot attempt. Arbitrarily changing hardware config on every
reboot is being gratuitously nasty to guest admins. The example about
causing Windows to require license reaactivation is on its own, enough
of a reason to ensure stable hardware once initial provisioning is
done.


> This isn't all we've thrown away, though. Libvirt also gave us a default
> machine type. When we create a new domain we'll get a new default machine
> type. If libvirt has been upgraded, eg during host maintenance, this isn't
> necessarily what it was before. Again, this can make our guests sad. Same
> goes for CPU model, default devices, and probably many more things I
> haven't thought of.

Yes indeed.

> Also... we lost the storage configuration of the guest: the information I
> propose to persist in persistent instance storage metadata.
> 
> We could store all of this information in Nova, but with the possible
> exception of storage metadata it really isn't at the level of 'management':
> it's the minutia of the hypervisor. In order to persist all of these things
> in Nova we'd have to implement them explicitly, and when libvirt/kvm grows
> more stuff we'll have to do that too. We'll need to mirror the
> functionality of libvirt in Nova, feature for feature. This is a red flag
> for me, and I think it means we should switch to libvirt being canonical.
> 
> I think we should be able to create a domain, but once created we should
> never redefine a domain. We can do adding and removing devices dynamically
> using libvirt's apis, secure in the knowledge that libvirt will persist
> this for us. When we upgrade the host, libvirt can ensure we don't break
> guests which are on it. Evacuate should be pretty much the only reason to
> start again.

And in fact we do persist the guest XML with libvirt already. We sadly
never use that info though - we just blindly overwrite it every time
with newly generated XML.

Fixing this should not be technically difficult for the most part.

> I raised this in the live migration sub-team meeting, and the immediate
> response was understandably conservative. I think this solves more problems
> than it creates, though, and it would result in Nova's libvirt driver
> getting a bit smaller and a bit simpler. That's a big win in my book.

I don't think it'll get significantly smaller/simpler, but it will
definitely be more intelligent and user friendly to do this IMHO.
As mentioned above, I think the windows license reactivation issue
alone is enough of a reason todo this.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org 

[openstack-dev] [nova][libvirt] Lets make libvirt's domain XML canonical

[Matthew Booth]

Currently the libvirt driver (mostly) considers the nova db canonical. That
is, we can throw away libvirt's domain XML at any time and recreate it from
Nova. Anywhere that doesn't assume this is a bug, because whatever
direction we choose we don't need 2 different sources of truth. The
thinking behind this is that we should always know what we told libvirt,
and if we lose that information then that's a bug.

This is true to a degree, and it's the reason I proposed the persistent
instance storage metadata spec: we lose track of how we configured an
instance's storage. I realised recently that this isn't the whole story,
though. Libvirt also automatically creates a bunch of state for us which we
didn't specify explicitly. We lose this every time we drop it and recreate.
For example, consider device addressing and ordering:

$ nova boot ...

We tell libvirt to give us a root disk, config disk, and a memballoon
device (amongst other things).

Libvirt assigns pci addresses to all of these things.

$ nova volume-attach ...

We tell libvirt to create a new disk attached to the given volume.

Libvirt assigns it a pci address.

$ nova reboot

We throw away libvirt's domain xml and create a new one from scratch.

Libvirt assigns new addresses for all of these devices.

Before reboot, the device order was: root disk, config disk, memballoon,
volume. After reboot the device order is: root disk, volume, config disk,
memballoon. Not only have all our devices changed address, which makes
Windows sad and paranoid about its licensing, and causes it to offline
volumes under certain circumstances, but our disks have been reordered.

This isn't all we've thrown away, though. Libvirt also gave us a default
machine type. When we create a new domain we'll get a new default machine
type. If libvirt has been upgraded, eg during host maintenance, this isn't
necessarily what it was before. Again, this can make our guests sad. Same
goes for CPU model, default devices, and probably many more things I
haven't thought of.

Also... we lost the storage configuration of the guest: the information I
propose to persist in persistent instance storage metadata.

We could store all of this information in Nova, but with the possible
exception of storage metadata it really isn't at the level of 'management':
it's the minutia of the hypervisor. In order to persist all of these things
in Nova we'd have to implement them explicitly, and when libvirt/kvm grows
more stuff we'll have to do that too. We'll need to mirror the
functionality of libvirt in Nova, feature for feature. This is a red flag
for me, and I think it means we should switch to libvirt being canonical.

I think we should be able to create a domain, but once created we should
never redefine a domain. We can do adding and removing devices dynamically
using libvirt's apis, secure in the knowledge that libvirt will persist
this for us. When we upgrade the host, libvirt can ensure we don't break
guests which are on it. Evacuate should be pretty much the only reason to
start again.

This would potentially obsolete my persistent instance metadata spec, and
the libvirt stable rescue spec, as well as this one:
https://review.openstack.org/#/c/347161/ .

I raised this in the live migration sub-team meeting, and the immediate
response was understandably conservative. I think this solves more problems
than it creates, though, and it would result in Nova's libvirt driver
getting a bit smaller and a bit simpler. That's a big win in my book.

Matt
-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev