Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On 02/03/2017 11:11 AM, Daniel P. Berrange wrote: On Fri, Feb 03, 2017 at 10:41:12AM +0100, Maxime Coquelin wrote: On 02/03/2017 10:27 AM, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote: On Thu, Feb 02, 2017 at 06:21:55PM +, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote: On Thu, Feb 02, 2017 at 05:29:08PM +, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: On Thu, Feb 02, 2017 at 05:10:28PM +, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: On Thu, Feb 02, 2017 at 03:06:21PM +, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config. eg those things in get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML. If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt. I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve. I don't think they currently understand the issues involved in cross-version migration though. This is a complex subject and easy to get wrong. It would be significantly better to figure it out at libvirt level since it already deals with cross-version migration issues. Libvirt considers vhost-user to be a blackbox though - it just exposes a UNIX socket, and whatever is on the other end is completely opaque. The fact that the other end might plumb the data stream into openvswitch is not something libvirt should know, as we don't want to end up having to add custom code to libvirt for every different vhost-user server impl. IOW, if the version str can be passed to QEMU, and then onto vhost-user backend in QEMU, then libvirt can be involved. If the version str has to be given to openvswitch that's not for libvirt to deall with. Regards, Daniel It's not just about passing it to QEMU. The following is needed: - need to query version when creating VM/device - need to query supported versions when migrating VM Those are both things that nova can do, since it knows what the vhost-user device in question is connected to, and so can query the versions, and check versions before triggering migration with libvirt It can, but then it will need to query libvirt or source for the version string since it's in the XML. No, it wouldn't be in the XML at all. Nova on the source queries what vhostuser version it has and what the target host has, and can prevent the migration if they're incompatible. This is not sufficient. Exactly the same as qemu machine type, this must be preserved from time of install and moved wherever VM goes. Nova has the ability todo that. I dont think libvirt has to be involved at all for this, as all the info can be obtained by nova/os-vif from the vhostuser impl it has configured. Given we are already confused at libvirt level, I find it highly unlikely upper levels will do the right thing. The only confusion is about what must consume the data. I mistakenly thought it was being proposed that the qemu vhostuser backend was being given a version string. Now it seems the version info must be set against the 3rd party component that vhostuser connects to, and that is outside scope of libvirt. That is entirely managed by Nova today, so Nova is the right thing to manage this versioning info. This is my understanding, with as example using Nova: 1. Before starting the VM, Nova queries source host's OVS and all possible destination hosts' OVS to retrieve supported versions 2. Nova chooses the first common version supported by all hosts 3. Nova create the dpdkvhostuser interfaces in OVS with passing the selected version as parameter That should be enough, right? Or maybe I miss something? It isn't realistic to check all possible destination hosts when first starting QEMU, not least becasue they may not yet exist. New hardware may be deployed *after* QEMU is first started. When starting the VM, it should just default to using whatever the latest version is on the host in question. There can be an optional host configuration parameter that admins can set to
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On Fri, Feb 03, 2017 at 10:41:12AM +0100, Maxime Coquelin wrote: > > > On 02/03/2017 10:27 AM, Daniel P. Berrange wrote: > > On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote: > > > On Thu, Feb 02, 2017 at 06:21:55PM +, Daniel P. Berrange wrote: > > > > On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote: > > > > > On Thu, Feb 02, 2017 at 05:29:08PM +, Daniel P. Berrange wrote: > > > > > > On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: > > > > > > > On Thu, Feb 02, 2017 at 05:10:28PM +, Daniel P. Berrange > > > > > > > wrote: > > > > > > > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin > > > > > > > > wrote: > > > > > > > > > On Thu, Feb 02, 2017 at 03:06:21PM +, Daniel P. Berrange > > > > > > > > > wrote: > > > > > > > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > > > > > > > > > > > > > > > > > > > > > It depends where / how in OVS it needs to be set. The > > > > > > > > > > > > only stuff libvirt > > > > > > > > > > > > does with OVS is to run 'add-port' and 'del-port' > > > > > > > > > > > > commands via the ovs > > > > > > > > > > > > cli tool. We pass through arguments from the port > > > > > > > > > > > > profile stored in the > > > > > > > > > > > > XML config. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > eg those things in get passed as cli args > > > > > > > > > > > > to the 'add-port' > > > > > > > > > > > > command. Soo if add-port needs this new version string, > > > > > > > > > > > > then we'd need > > > > > > > > > > > > to add the version to the openvswitch virtualport XML. > > > > > > > > > > > > > > > > > > > > > > > > If the version is provided to OVS in a different > > > > > > > > > > > > command, then it would > > > > > > > > > > > > probably be outside scope of libvirt. > > > > > > > > > > > > > > > > > > > > > > I think it would make sense to be a parameter of the > > > > > > > > > > > add-port command. > > > > > > > > > > > But it would be for vhost-user related add-port command, > > > > > > > > > > > I didn't find > > > > > > > > > > > where/if this is managed in libvirt XML. > > > > > > > > > > > > > > > > > > > > For vhost-user, libvirt does not have any interaction with > > > > > > > > > > OVS at > > > > > > > > > > all. If the thing that's using the vhost-user UNIX socket, > > > > > > > > > > in turn > > > > > > > > > > connects to OVS, that's outside scope of libvirt. IOW, for > > > > > > > > > > vhost-user > > > > > > > > > > OVS it seems like that job is for Nova / os-vif to solve. > > > > > > > > > > > > > > > > > > I don't think they currently understand the issues involved in > > > > > > > > > cross-version migration though. This is a complex subject > > > > > > > > > and easy to > > > > > > > > > get wrong. It would be significantly better to figure it out > > > > > > > > > at libvirt > > > > > > > > > level since it already deals with cross-version migration > > > > > > > > > issues. > > > > > > > > > > > > > > > > Libvirt considers vhost-user to be a blackbox though - it just > > > > > > > > exposes > > > > > > > > a UNIX socket, and whatever is on the other end is completely > > > > > > > > opaque. > > > > > > > > The fact that the other end might plumb the data stream into > > > > > > > > openvswitch > > > > > > > > is not something libvirt should know, as we don't want to end > > > > > > > > up having > > > > > > > > to add custom code to libvirt for every different vhost-user > > > > > > > > server > > > > > > > > impl. > > > > > > > > > > > > > > > > IOW, if the version str can be passed to QEMU, and then onto > > > > > > > > vhost-user > > > > > > > > backend in QEMU, then libvirt can be involved. If the version > > > > > > > > str has > > > > > > > > to be given to openvswitch that's not for libvirt to deall with. > > > > > > > > > > > > > > > > Regards, > > > > > > > > Daniel > > > > > > > > > > > > > > It's not just about passing it to QEMU. The following is needed: > > > > > > > - need to query version when creating VM/device > > > > > > > - need to query supported versions when migrating VM > > > > > > > > > > > > Those are both things that nova can do, since it knows what the > > > > > > vhost-user > > > > > > device in question is connected to, and so can query the versions, > > > > > > and check > > > > > > versions before triggering migration with libvirt > > > > > > > > > > It can, but then it will need to query libvirt or source for the > > > > > version > > > > > string since it's in
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On 02/03/2017 10:27 AM, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote: On Thu, Feb 02, 2017 at 06:21:55PM +, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote: On Thu, Feb 02, 2017 at 05:29:08PM +, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: On Thu, Feb 02, 2017 at 05:10:28PM +, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: On Thu, Feb 02, 2017 at 03:06:21PM +, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. We pass through arguments from the port profile stored in the XML config. eg those things in get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML. If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt. I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve. I don't think they currently understand the issues involved in cross-version migration though. This is a complex subject and easy to get wrong. It would be significantly better to figure it out at libvirt level since it already deals with cross-version migration issues. Libvirt considers vhost-user to be a blackbox though - it just exposes a UNIX socket, and whatever is on the other end is completely opaque. The fact that the other end might plumb the data stream into openvswitch is not something libvirt should know, as we don't want to end up having to add custom code to libvirt for every different vhost-user server impl. IOW, if the version str can be passed to QEMU, and then onto vhost-user backend in QEMU, then libvirt can be involved. If the version str has to be given to openvswitch that's not for libvirt to deall with. Regards, Daniel It's not just about passing it to QEMU. The following is needed: - need to query version when creating VM/device - need to query supported versions when migrating VM Those are both things that nova can do, since it knows what the vhost-user device in question is connected to, and so can query the versions, and check versions before triggering migration with libvirt It can, but then it will need to query libvirt or source for the version string since it's in the XML. No, it wouldn't be in the XML at all. Nova on the source queries what vhostuser version it has and what the target host has, and can prevent the migration if they're incompatible. This is not sufficient. Exactly the same as qemu machine type, this must be preserved from time of install and moved wherever VM goes. Nova has the ability todo that. I dont think libvirt has to be involved at all for this, as all the info can be obtained by nova/os-vif from the vhostuser impl it has configured. Given we are already confused at libvirt level, I find it highly unlikely upper levels will do the right thing. The only confusion is about what must consume the data. I mistakenly thought it was being proposed that the qemu vhostuser backend was being given a version string. Now it seems the version info must be set against the 3rd party component that vhostuser connects to, and that is outside scope of libvirt. That is entirely managed by Nova today, so Nova is the right thing to manage this versioning info. This is my understanding, with as example using Nova: 1. Before starting the VM, Nova queries source host's OVS and all possible destination hosts' OVS to retrieve supported versions 2. Nova chooses the first common version supported by all hosts 3. Nova create the dpdkvhostuser interfaces in OVS with passing the selected version as parameter That should be enough, right? Or maybe I miss something? Thanks, Maxime ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On Thu, Feb 02, 2017 at 08:27:18PM +0200, Michael S. Tsirkin wrote: > On Thu, Feb 02, 2017 at 06:21:55PM +, Daniel P. Berrange wrote: > > On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote: > > > On Thu, Feb 02, 2017 at 05:29:08PM +, Daniel P. Berrange wrote: > > > > On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: > > > > > On Thu, Feb 02, 2017 at 05:10:28PM +, Daniel P. Berrange wrote: > > > > > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: > > > > > > > On Thu, Feb 02, 2017 at 03:06:21PM +, Daniel P. Berrange > > > > > > > wrote: > > > > > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > > > > > > > > > > > > > > > > > It depends where / how in OVS it needs to be set. The only > > > > > > > > > > stuff libvirt > > > > > > > > > > does with OVS is to run 'add-port' and 'del-port' commands > > > > > > > > > > via the ovs > > > > > > > > > > cli tool. We pass through arguments from the port profile > > > > > > > > > > stored in the > > > > > > > > > > XML config. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > eg those things in get passed as cli args to > > > > > > > > > > the 'add-port' > > > > > > > > > > command. Soo if add-port needs this new version string, > > > > > > > > > > then we'd need > > > > > > > > > > to add the version to the openvswitch virtualport XML. > > > > > > > > > > > > > > > > > > > > If the version is provided to OVS in a different command, > > > > > > > > > > then it would > > > > > > > > > > probably be outside scope of libvirt. > > > > > > > > > > > > > > > > > > I think it would make sense to be a parameter of the add-port > > > > > > > > > command. > > > > > > > > > But it would be for vhost-user related add-port command, I > > > > > > > > > didn't find > > > > > > > > > where/if this is managed in libvirt XML. > > > > > > > > > > > > > > > > For vhost-user, libvirt does not have any interaction with OVS > > > > > > > > at > > > > > > > > all. If the thing that's using the vhost-user UNIX socket, in > > > > > > > > turn > > > > > > > > connects to OVS, that's outside scope of libvirt. IOW, for > > > > > > > > vhost-user > > > > > > > > OVS it seems like that job is for Nova / os-vif to solve. > > > > > > > > > > > > > > I don't think they currently understand the issues involved in > > > > > > > cross-version migration though. This is a complex subject and > > > > > > > easy to > > > > > > > get wrong. It would be significantly better to figure it out at > > > > > > > libvirt > > > > > > > level since it already deals with cross-version migration issues. > > > > > > > > > > > > Libvirt considers vhost-user to be a blackbox though - it just > > > > > > exposes > > > > > > a UNIX socket, and whatever is on the other end is completely > > > > > > opaque. > > > > > > The fact that the other end might plumb the data stream into > > > > > > openvswitch > > > > > > is not something libvirt should know, as we don't want to end up > > > > > > having > > > > > > to add custom code to libvirt for every different vhost-user server > > > > > > impl. > > > > > > > > > > > > IOW, if the version str can be passed to QEMU, and then onto > > > > > > vhost-user > > > > > > backend in QEMU, then libvirt can be involved. If the version str > > > > > > has > > > > > > to be given to openvswitch that's not for libvirt to deall with. > > > > > > > > > > > > Regards, > > > > > > Daniel > > > > > > > > > > It's not just about passing it to QEMU. The following is needed: > > > > > - need to query version when creating VM/device > > > > > - need to query supported versions when migrating VM > > > > > > > > Those are both things that nova can do, since it knows what the > > > > vhost-user > > > > device in question is connected to, and so can query the versions, and > > > > check > > > > versions before triggering migration with libvirt > > > > > > It can, but then it will need to query libvirt or source for the version > > > string since it's in the XML. > > > > No, it wouldn't be in the XML at all. Nova on the source queries what > > vhostuser version it has and what the target host has, and can prevent > > the migration if they're incompatible. > > This is not sufficient. Exactly the same as qemu machine type, > this must be preserved from time of install and moved > wherever VM goes. Nova has the ability todo that. > > I dont think libvirt has to be > > involved at all for this, as all the info can be obtained by nova/os-vif > > from the vhostuser impl it has configured. > > Given we are already confused at libvirt level,
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On Thu, Feb 02, 2017 at 06:21:55PM +, Daniel P. Berrange wrote: > On Thu, Feb 02, 2017 at 07:31:49PM +0200, Michael S. Tsirkin wrote: > > On Thu, Feb 02, 2017 at 05:29:08PM +, Daniel P. Berrange wrote: > > > On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: > > > > On Thu, Feb 02, 2017 at 05:10:28PM +, Daniel P. Berrange wrote: > > > > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: > > > > > > On Thu, Feb 02, 2017 at 03:06:21PM +, Daniel P. Berrange wrote: > > > > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: > > > > > > > > > > > > > > > > > > > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > > > > > > > > > > > > > > > It depends where / how in OVS it needs to be set. The only > > > > > > > > > stuff libvirt > > > > > > > > > does with OVS is to run 'add-port' and 'del-port' commands > > > > > > > > > via the ovs > > > > > > > > > cli tool. We pass through arguments from the port profile > > > > > > > > > stored in the > > > > > > > > > XML config. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > eg those things in get passed as cli args to > > > > > > > > > the 'add-port' > > > > > > > > > command. Soo if add-port needs this new version string, then > > > > > > > > > we'd need > > > > > > > > > to add the version to the openvswitch virtualport XML. > > > > > > > > > > > > > > > > > > If the version is provided to OVS in a different command, > > > > > > > > > then it would > > > > > > > > > probably be outside scope of libvirt. > > > > > > > > > > > > > > > > I think it would make sense to be a parameter of the add-port > > > > > > > > command. > > > > > > > > But it would be for vhost-user related add-port command, I > > > > > > > > didn't find > > > > > > > > where/if this is managed in libvirt XML. > > > > > > > > > > > > > > For vhost-user, libvirt does not have any interaction with OVS at > > > > > > > all. If the thing that's using the vhost-user UNIX socket, in turn > > > > > > > connects to OVS, that's outside scope of libvirt. IOW, for > > > > > > > vhost-user > > > > > > > OVS it seems like that job is for Nova / os-vif to solve. > > > > > > > > > > > > I don't think they currently understand the issues involved in > > > > > > cross-version migration though. This is a complex subject and easy > > > > > > to > > > > > > get wrong. It would be significantly better to figure it out at > > > > > > libvirt > > > > > > level since it already deals with cross-version migration issues. > > > > > > > > > > Libvirt considers vhost-user to be a blackbox though - it just exposes > > > > > a UNIX socket, and whatever is on the other end is completely opaque. > > > > > The fact that the other end might plumb the data stream into > > > > > openvswitch > > > > > is not something libvirt should know, as we don't want to end up > > > > > having > > > > > to add custom code to libvirt for every different vhost-user server > > > > > impl. > > > > > > > > > > IOW, if the version str can be passed to QEMU, and then onto > > > > > vhost-user > > > > > backend in QEMU, then libvirt can be involved. If the version str has > > > > > to be given to openvswitch that's not for libvirt to deall with. > > > > > > > > > > Regards, > > > > > Daniel > > > > > > > > It's not just about passing it to QEMU. The following is needed: > > > > - need to query version when creating VM/device > > > > - need to query supported versions when migrating VM > > > > > > Those are both things that nova can do, since it knows what the vhost-user > > > device in question is connected to, and so can query the versions, and > > > check > > > versions before triggering migration with libvirt > > > > It can, but then it will need to query libvirt or source for the version > > string since it's in the XML. > > No, it wouldn't be in the XML at all. Nova on the source queries what > vhostuser version it has and what the target host has, and can prevent > the migration if they're incompatible. This is not sufficient. Exactly the same as qemu machine type, this must be preserved from time of install and moved wherever VM goes. > I dont think libvirt has to be > involved at all for this, as all the info can be obtained by nova/os-vif > from the vhostuser impl it has configured. > > Regards, > Daniel Given we are already confused at libvirt level, I find it highly unlikely upper levels will do the right thing. Generally I simply don't understand why would we expose to higher levels something which is fundamentally not a policy decision. > -- > |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| > |: http://libvirt.org -o- http://virt-manager.org :| > |:
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On Thu, Feb 02, 2017 at 05:29:08PM +, Daniel P. Berrange wrote: > On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: > > On Thu, Feb 02, 2017 at 05:10:28PM +, Daniel P. Berrange wrote: > > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: > > > > On Thu, Feb 02, 2017 at 03:06:21PM +, Daniel P. Berrange wrote: > > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: > > > > > > > > > > > > > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > > > > > > > > > > > It depends where / how in OVS it needs to be set. The only stuff > > > > > > > libvirt > > > > > > > does with OVS is to run 'add-port' and 'del-port' commands via > > > > > > > the ovs > > > > > > > cli tool. We pass through arguments from the port profile stored > > > > > > > in the > > > > > > > XML config. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > eg those things in get passed as cli args to the > > > > > > > 'add-port' > > > > > > > command. Soo if add-port needs this new version string, then we'd > > > > > > > need > > > > > > > to add the version to the openvswitch virtualport XML. > > > > > > > > > > > > > > If the version is provided to OVS in a different command, then it > > > > > > > would > > > > > > > probably be outside scope of libvirt. > > > > > > > > > > > > I think it would make sense to be a parameter of the add-port > > > > > > command. > > > > > > But it would be for vhost-user related add-port command, I didn't > > > > > > find > > > > > > where/if this is managed in libvirt XML. > > > > > > > > > > For vhost-user, libvirt does not have any interaction with OVS at > > > > > all. If the thing that's using the vhost-user UNIX socket, in turn > > > > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user > > > > > OVS it seems like that job is for Nova / os-vif to solve. > > > > > > > > I don't think they currently understand the issues involved in > > > > cross-version migration though. This is a complex subject and easy to > > > > get wrong. It would be significantly better to figure it out at libvirt > > > > level since it already deals with cross-version migration issues. > > > > > > Libvirt considers vhost-user to be a blackbox though - it just exposes > > > a UNIX socket, and whatever is on the other end is completely opaque. > > > The fact that the other end might plumb the data stream into openvswitch > > > is not something libvirt should know, as we don't want to end up having > > > to add custom code to libvirt for every different vhost-user server > > > impl. > > > > > > IOW, if the version str can be passed to QEMU, and then onto vhost-user > > > backend in QEMU, then libvirt can be involved. If the version str has > > > to be given to openvswitch that's not for libvirt to deall with. > > > > > > Regards, > > > Daniel > > > > It's not just about passing it to QEMU. The following is needed: > > - need to query version when creating VM/device > > - need to query supported versions when migrating VM > > Those are both things that nova can do, since it knows what the vhost-user > device in question is connected to, and so can query the versions, and check > versions before triggering migration with libvirt > > Regards, > Daniel It can, but then it will need to query libvirt or source for the version string since it's in the XML. > -- > |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| > |: http://libvirt.org -o- http://virt-manager.org :| > |: http://entangle-photo.org -o-http://search.cpan.org/~danberr/ :| ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On Thu, Feb 02, 2017 at 07:20:35PM +0200, Michael S. Tsirkin wrote: > On Thu, Feb 02, 2017 at 05:10:28PM +, Daniel P. Berrange wrote: > > On Thu, Feb 02, 2017 at 06:21:55PM +0200, Michael S. Tsirkin wrote: > > > On Thu, Feb 02, 2017 at 03:06:21PM +, Daniel P. Berrange wrote: > > > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: > > > > > > > > > > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > > > > > > > > > It depends where / how in OVS it needs to be set. The only stuff > > > > > > libvirt > > > > > > does with OVS is to run 'add-port' and 'del-port' commands via the > > > > > > ovs > > > > > > cli tool. We pass through arguments from the port profile stored in > > > > > > the > > > > > > XML config. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > > > > > > > > > > > > > > > > > > > > > > > eg those things in get passed as cli args to the > > > > > > 'add-port' > > > > > > command. Soo if add-port needs this new version string, then we'd > > > > > > need > > > > > > to add the version to the openvswitch virtualport XML. > > > > > > > > > > > > If the version is provided to OVS in a different command, then it > > > > > > would > > > > > > probably be outside scope of libvirt. > > > > > > > > > > I think it would make sense to be a parameter of the add-port command. > > > > > But it would be for vhost-user related add-port command, I didn't find > > > > > where/if this is managed in libvirt XML. > > > > > > > > For vhost-user, libvirt does not have any interaction with OVS at > > > > all. If the thing that's using the vhost-user UNIX socket, in turn > > > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user > > > > OVS it seems like that job is for Nova / os-vif to solve. > > > > > > I don't think they currently understand the issues involved in > > > cross-version migration though. This is a complex subject and easy to > > > get wrong. It would be significantly better to figure it out at libvirt > > > level since it already deals with cross-version migration issues. > > > > Libvirt considers vhost-user to be a blackbox though - it just exposes > > a UNIX socket, and whatever is on the other end is completely opaque. > > The fact that the other end might plumb the data stream into openvswitch > > is not something libvirt should know, as we don't want to end up having > > to add custom code to libvirt for every different vhost-user server > > impl. > > > > IOW, if the version str can be passed to QEMU, and then onto vhost-user > > backend in QEMU, then libvirt can be involved. If the version str has > > to be given to openvswitch that's not for libvirt to deall with. > > > > Regards, > > Daniel > > It's not just about passing it to QEMU. The following is needed: > - need to query version when creating VM/device > - need to query supported versions when migrating VM Those are both things that nova can do, since it knows what the vhost-user device in question is connected to, and so can query the versions, and check versions before triggering migration with libvirt Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o-http://search.cpan.org/~danberr/ :| ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On 02/02/2017 06:09 PM, Michael S. Tsirkin wrote: On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote: On 02/02/2017 10:06 AM, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. (aside note: the code that exec's ovs-vsctl was written back when there was no standardized API for performing such operations. libvirt would prefer to not exec external programs though, and I've heard that OVS may now have an official API of some sort for doing things like this (maybe via netlink or dbus or something?) If that's the case, can someone point me in the right direction?) We pass through arguments from the port profile stored in the XML config. eg those things in get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML. If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt. I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve. This brings up another tangentially related question I came up against last night - qemu now has an option to report the host's MTU to the guest for virtio and vhost-user interfaces, and Michal Privoznik recently pushed patches to set the MTU sent to the guest via an explicit in libvirt's interface config: https://bugzilla.redhat.com/1408701 But it would be much nicer if libvirt could learn the MTU of [that stuff at the other end of the unix socket] without requiring intervention in libvirt's config. For example, I'm just now testing patches for tap-based interfaces (connecting to Linux host bridges or OVS switches) that query the current MTU of the bridge and report that to qemu; this eliminates the burden of configuring each interface of each guest individually (and changing that config in all those places if someone ever wants to change the MTU of the bridge). As Dan says, though, libvirt's only interaction in the case of vhost-user is with the unix socket. Is there any way to learn what is the appropriate MTU from OVS in these cases? Or must Nova (or ovirt or some poor user) set that up in the libvirt config for every single interface? We could add commands for all kind of queries to the vhost-user protocol. libvirt would have to learn the vhost-user protocol though. Interested? I think it could be possible to query the MTU value from the OVS DB using its JSON RPC-like API, but this is something I haven't tried. I guess it would need to resolve the ovs interface from the vhost-user socket path. Can people familiar with OVS confirm this is something possible? Regards, Maxime ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On Thu, Feb 02, 2017 at 11:47:57AM -0500, Laine Stump wrote: > On 02/02/2017 10:06 AM, Daniel P. Berrange wrote: > > On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: > > > > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt > > > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs > > > > cli tool. > > (aside note: the code that exec's ovs-vsctl was written back when there was > no standardized API for performing such operations. libvirt would prefer to > not exec external programs though, and I've heard that OVS may now have an > official API of some sort for doing things like this (maybe via netlink or > dbus or something?) If that's the case, can someone point me in the right > direction?) > > > > > We pass through arguments from the port profile stored in the > > > > XML config. > > > > > > > > > > > > > > > > > > > > > > > interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > > > > > > > > > > > > > > > eg those things in get passed as cli args to the > > > > 'add-port' > > > > command. Soo if add-port needs this new version string, then we'd need > > > > to add the version to the openvswitch virtualport XML. > > > > > > > > If the version is provided to OVS in a different command, then it would > > > > probably be outside scope of libvirt. > > > I think it would make sense to be a parameter of the add-port command. > > > But it would be for vhost-user related add-port command, I didn't find > > > where/if this is managed in libvirt XML. > > For vhost-user, libvirt does not have any interaction with OVS at > > all. If the thing that's using the vhost-user UNIX socket, in turn > > connects to OVS, that's outside scope of libvirt. IOW, for vhost-user > > OVS it seems like that job is for Nova / os-vif to solve. > > This brings up another tangentially related question I came up against last > night - qemu now has an option to report the host's MTU to the guest for > virtio and vhost-user interfaces, and Michal Privoznik recently pushed > patches to set the MTU sent to the guest via an explicit in > libvirt's interface config: > >https://bugzilla.redhat.com/1408701 > > But it would be much nicer if libvirt could learn the MTU of [that stuff at > the other end of the unix socket] without requiring intervention in > libvirt's config. For example, I'm just now testing patches for tap-based > interfaces (connecting to Linux host bridges or OVS switches) that query the > current MTU of the bridge and report that to qemu; this eliminates the > burden of configuring each interface of each guest individually (and > changing that config in all those places if someone ever wants to change the > MTU of the bridge). > > As Dan says, though, libvirt's only interaction in the case of vhost-user is > with the unix socket. Is there any way to learn what is the appropriate MTU > from OVS in these cases? Or must Nova (or ovirt or some poor user) set that > up in the libvirt config for every single interface? We could add commands for all kind of queries to the vhost-user protocol. libvirt would have to learn the vhost-user protocol though. Interested? -- MST ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On 02/02/2017 10:06 AM, Daniel P. Berrange wrote: On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: It depends where / how in OVS it needs to be set. The only stuff libvirt does with OVS is to run 'add-port' and 'del-port' commands via the ovs cli tool. (aside note: the code that exec's ovs-vsctl was written back when there was no standardized API for performing such operations. libvirt would prefer to not exec external programs though, and I've heard that OVS may now have an official API of some sort for doing things like this (maybe via netlink or dbus or something?) If that's the case, can someone point me in the right direction?) We pass through arguments from the port profile stored in the XML config. eg those things in get passed as cli args to the 'add-port' command. Soo if add-port needs this new version string, then we'd need to add the version to the openvswitch virtualport XML. If the version is provided to OVS in a different command, then it would probably be outside scope of libvirt. I think it would make sense to be a parameter of the add-port command. But it would be for vhost-user related add-port command, I didn't find where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve. This brings up another tangentially related question I came up against last night - qemu now has an option to report the host's MTU to the guest for virtio and vhost-user interfaces, and Michal Privoznik recently pushed patches to set the MTU sent to the guest via an explicit in libvirt's interface config: https://bugzilla.redhat.com/1408701 But it would be much nicer if libvirt could learn the MTU of [that stuff at the other end of the unix socket] without requiring intervention in libvirt's config. For example, I'm just now testing patches for tap-based interfaces (connecting to Linux host bridges or OVS switches) that query the current MTU of the bridge and report that to qemu; this eliminates the burden of configuring each interface of each guest individually (and changing that config in all those places if someone ever wants to change the MTU of the bridge). As Dan says, though, libvirt's only interaction in the case of vhost-user is with the unix socket. Is there any way to learn what is the appropriate MTU from OVS in these cases? Or must Nova (or ovirt or some poor user) set that up in the libvirt config for every single interface? ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On Thu, Feb 02, 2017 at 03:14:01PM +0100, Maxime Coquelin wrote: > > > On 02/01/2017 12:41 PM, Daniel P. Berrange wrote: > > > > It depends where / how in OVS it needs to be set. The only stuff libvirt > > does with OVS is to run 'add-port' and 'del-port' commands via the ovs > > cli tool. We pass through arguments from the port profile stored in the > > XML config. > > > > > > > > > >> interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/> > > > > > > > > eg those things in get passed as cli args to the 'add-port' > > command. Soo if add-port needs this new version string, then we'd need > > to add the version to the openvswitch virtualport XML. > > > > If the version is provided to OVS in a different command, then it would > > probably be outside scope of libvirt. > > I think it would make sense to be a parameter of the add-port command. > But it would be for vhost-user related add-port command, I didn't find > where/if this is managed in libvirt XML. For vhost-user, libvirt does not have any interaction with OVS at all. If the thing that's using the vhost-user UNIX socket, in turn connects to OVS, that's outside scope of libvirt. IOW, for vhost-user OVS it seems like that job is for Nova / os-vif to solve. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o-http://search.cpan.org/~danberr/ :| ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On Wed, Feb 01, 2017 at 11:41:19AM +, Daniel P. Berrange wrote: > On Wed, Feb 01, 2017 at 12:33:22PM +0100, Maxime Coquelin wrote: > > > > > > On 02/01/2017 10:43 AM, Daniel P. Berrange wrote: > > > On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote: > > > > On 02/01/2017 09:35 AM, Maxime Coquelin wrote: > > > > > > > > Solution 3: Libvirt queries OVS for vhost backend version string: *OK* > > > > > == > > > > > > > > > > > > > > > The idea is to have a table of supported versions, associated to > > > > > key/value pairs. Libvirt could query the list of supported versions > > > > > strings for each hosts, and select the first common one among all > > > > > hosts. > > > > > > > > How does libvirt know what hosts to ask? Libvirt aims on managing a > > > > single host. It has no knowledge of other hosts on the network. That's > > > > task for upper layers like RHEV, OpenStack, etc. > > > > > > > > > > > > > > Then, libvirt would ask OVS to probe the vhost-user interfaces in the > > > > > selected version (compatibility mode). For example host A runs > > > > > OVS-2.7, > > > > > and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode > > > > > (e.g. with indirect descriptors disabled), which should be selected at > > > > > vhost-user interface probe time. > > > > > > > > > > Advantage of doing so is that libvirt does not need any update if new > > > > > keys are introduced (i.e. it does not need to know how the new keys > > > > > have > > > > > to be handled), all these checks remain in OVS's vhost-user > > > > > implementation. > > > > > > > > And that's where they should stay. Duplicating code between projects > > > > will inevitably lead to a divergence. > > > > > > > > > > > > > > Ideally, we would support per vhost-user interface compatibility > > > > > mode, > > > > > which may have an impact also on DPDK API, as the Virtio feature > > > > > update > > > > > API is global, and not per port. > > > > > > > > In general, I don't think we want any kind of this logic in libvirt. > > > > Either: > > > > > > > > a) fallback logic should be implemented in qemu (e.g. upon migration it > > > > should detect that the migrated guest uses certain version and thus set > > > > backend to use that version or error out and cancel migration), or > > > > > > > > b) libvirt would grew another element/attribute to specify version of > > > > vhost-user backend in use and do nothing more than pass it to qemu. At > > > > the same time, we can provide an API (or extend and existing one, e.g. > > > > virsh domcapabilities) to list all available versions on given host. > > > > Upper layer, which knows what are the possible hosts suitable for > > > > virtualization, can then use this API to ask all the hosts, construct > > > > the matrix, select preferred version and put it into libvirt's domain > > > > XML. > > > > > > > > But frankly, I don't like b) that much. Lets put the fact this is OVS > > > > aside for a moment. Just pretend this is a generic device in qemu. Would > > > > we do the same magic with it? No! Or lets talk about machine types. You > > > > spawn -M type$((X+1)) guest and then decide to migrate it to a host with > > > > older qemu wich supports just typeX. Well, you get an error. Do we care? > > > > Not at all! It's your responsibility (as user/admin) to upgrade the qemu > > > > so that it supports new machine type. I think the same applies to OVS. > > > > > > With machine types, if the latest machine type is X, libvirt allows > > > the mgmt app to spawn a guest with mcahine type X-1, so that you can > > > later migrate the VM to a host with older QEMU. > > > > > > With the vhost user device, the VM will always be launched with version > > > Y. There's currently no way to request launching the vhost user device > > > wtih version Y-1. So even if you set your machine type to X-1 for > > > compat with older host, vhost user will be stuck at version Y preventing > > > the migration. > > > > > > One argument would be to say that the vhost user featureset should be > > > determined by the machine type version, instead of introducing a new > > > version. The complexity is that vhost-user is a pretty dumb device > > > from QEMUs POV - most of the interesting logic & the features that > > > need to be constrained lie in code outside of QEMU, in whatever > > > server is connected to the vhost user socket. > > > > > > So I can see the value in allowing a simple version string to be > > > associated with the vhost-user NIC. > > > > > > What I'm unclear about is how we would be able to report capabilities > > > for a host to enumerate what versions were possible. Libvirt doesn't > > > interact with any of the 3rd party vhost user servers, so it is probably > > > out of scope - it'd be upto the higher level mgmt app to talk to DPDK > > > and figure out what versions it supports. > > > > > >
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On Wed, Feb 01, 2017 at 12:33:22PM +0100, Maxime Coquelin wrote: > > > On 02/01/2017 10:43 AM, Daniel P. Berrange wrote: > > On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote: > > > On 02/01/2017 09:35 AM, Maxime Coquelin wrote: > > > > > > Solution 3: Libvirt queries OVS for vhost backend version string: *OK* > > > > == > > > > > > > > > > > > The idea is to have a table of supported versions, associated to > > > > key/value pairs. Libvirt could query the list of supported versions > > > > strings for each hosts, and select the first common one among all hosts. > > > > > > How does libvirt know what hosts to ask? Libvirt aims on managing a > > > single host. It has no knowledge of other hosts on the network. That's > > > task for upper layers like RHEV, OpenStack, etc. > > > > > > > > > > > Then, libvirt would ask OVS to probe the vhost-user interfaces in the > > > > selected version (compatibility mode). For example host A runs OVS-2.7, > > > > and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode > > > > (e.g. with indirect descriptors disabled), which should be selected at > > > > vhost-user interface probe time. > > > > > > > > Advantage of doing so is that libvirt does not need any update if new > > > > keys are introduced (i.e. it does not need to know how the new keys have > > > > to be handled), all these checks remain in OVS's vhost-user > > > > implementation. > > > > > > And that's where they should stay. Duplicating code between projects > > > will inevitably lead to a divergence. > > > > > > > > > > > Ideally, we would support per vhost-user interface compatibility mode, > > > > which may have an impact also on DPDK API, as the Virtio feature update > > > > API is global, and not per port. > > > > > > In general, I don't think we want any kind of this logic in libvirt. > > > Either: > > > > > > a) fallback logic should be implemented in qemu (e.g. upon migration it > > > should detect that the migrated guest uses certain version and thus set > > > backend to use that version or error out and cancel migration), or > > > > > > b) libvirt would grew another element/attribute to specify version of > > > vhost-user backend in use and do nothing more than pass it to qemu. At > > > the same time, we can provide an API (or extend and existing one, e.g. > > > virsh domcapabilities) to list all available versions on given host. > > > Upper layer, which knows what are the possible hosts suitable for > > > virtualization, can then use this API to ask all the hosts, construct > > > the matrix, select preferred version and put it into libvirt's domain XML. > > > > > > But frankly, I don't like b) that much. Lets put the fact this is OVS > > > aside for a moment. Just pretend this is a generic device in qemu. Would > > > we do the same magic with it? No! Or lets talk about machine types. You > > > spawn -M type$((X+1)) guest and then decide to migrate it to a host with > > > older qemu wich supports just typeX. Well, you get an error. Do we care? > > > Not at all! It's your responsibility (as user/admin) to upgrade the qemu > > > so that it supports new machine type. I think the same applies to OVS. > > > > With machine types, if the latest machine type is X, libvirt allows > > the mgmt app to spawn a guest with mcahine type X-1, so that you can > > later migrate the VM to a host with older QEMU. > > > > With the vhost user device, the VM will always be launched with version > > Y. There's currently no way to request launching the vhost user device > > wtih version Y-1. So even if you set your machine type to X-1 for > > compat with older host, vhost user will be stuck at version Y preventing > > the migration. > > > > One argument would be to say that the vhost user featureset should be > > determined by the machine type version, instead of introducing a new > > version. The complexity is that vhost-user is a pretty dumb device > > from QEMUs POV - most of the interesting logic & the features that > > need to be constrained lie in code outside of QEMU, in whatever > > server is connected to the vhost user socket. > > > > So I can see the value in allowing a simple version string to be > > associated with the vhost-user NIC. > > > > What I'm unclear about is how we would be able to report capabilities > > for a host to enumerate what versions were possible. Libvirt doesn't > > interact with any of the 3rd party vhost user servers, so it is probably > > out of scope - it'd be upto the higher level mgmt app to talk to DPDK > > and figure out what versions it supports. > > > > That does make me wonder though if libvirt & QEMU need to be involved > > at all in any part. > > Indeed, if the higher level management tool decides for the VM's machine > type, it is where it should also be done for the vhost-user backend. I > now understand this does not make much sense to have libvirt
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On Wed, Feb 01, 2017 at 10:14:54AM +0100, Michal Privoznik wrote: > On 02/01/2017 09:35 AM, Maxime Coquelin wrote: > > Solution 3: Libvirt queries OVS for vhost backend version string: *OK* > > == > > > > > > The idea is to have a table of supported versions, associated to > > key/value pairs. Libvirt could query the list of supported versions > > strings for each hosts, and select the first common one among all hosts. > > How does libvirt know what hosts to ask? Libvirt aims on managing a > single host. It has no knowledge of other hosts on the network. That's > task for upper layers like RHEV, OpenStack, etc. > > > > > Then, libvirt would ask OVS to probe the vhost-user interfaces in the > > selected version (compatibility mode). For example host A runs OVS-2.7, > > and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode > > (e.g. with indirect descriptors disabled), which should be selected at > > vhost-user interface probe time. > > > > Advantage of doing so is that libvirt does not need any update if new > > keys are introduced (i.e. it does not need to know how the new keys have > > to be handled), all these checks remain in OVS's vhost-user implementation. > > And that's where they should stay. Duplicating code between projects > will inevitably lead to a divergence. > > > > > Ideally, we would support per vhost-user interface compatibility mode, > > which may have an impact also on DPDK API, as the Virtio feature update > > API is global, and not per port. > > In general, I don't think we want any kind of this logic in libvirt. Either: > > a) fallback logic should be implemented in qemu (e.g. upon migration it > should detect that the migrated guest uses certain version and thus set > backend to use that version or error out and cancel migration), or > > b) libvirt would grew another element/attribute to specify version of > vhost-user backend in use and do nothing more than pass it to qemu. At > the same time, we can provide an API (or extend and existing one, e.g. > virsh domcapabilities) to list all available versions on given host. > Upper layer, which knows what are the possible hosts suitable for > virtualization, can then use this API to ask all the hosts, construct > the matrix, select preferred version and put it into libvirt's domain XML. > > But frankly, I don't like b) that much. Lets put the fact this is OVS > aside for a moment. Just pretend this is a generic device in qemu. Would > we do the same magic with it? No! Or lets talk about machine types. You > spawn -M type$((X+1)) guest and then decide to migrate it to a host with > older qemu wich supports just typeX. Well, you get an error. Do we care? > Not at all! It's your responsibility (as user/admin) to upgrade the qemu > so that it supports new machine type. I think the same applies to OVS. With machine types, if the latest machine type is X, libvirt allows the mgmt app to spawn a guest with mcahine type X-1, so that you can later migrate the VM to a host with older QEMU. With the vhost user device, the VM will always be launched with version Y. There's currently no way to request launching the vhost user device wtih version Y-1. So even if you set your machine type to X-1 for compat with older host, vhost user will be stuck at version Y preventing the migration. One argument would be to say that the vhost user featureset should be determined by the machine type version, instead of introducing a new version. The complexity is that vhost-user is a pretty dumb device from QEMUs POV - most of the interesting logic & the features that need to be constrained lie in code outside of QEMU, in whatever server is connected to the vhost user socket. So I can see the value in allowing a simple version string to be associated with the vhost-user NIC. What I'm unclear about is how we would be able to report capabilities for a host to enumerate what versions were possible. Libvirt doesn't interact with any of the 3rd party vhost user servers, so it is probably out of scope - it'd be upto the higher level mgmt app to talk to DPDK and figure out what versions it supports. That does make me wonder though if libvirt & QEMU need to be involved at all in any part. When provisioning a new guest, the mgmt app presumably has to talk to DPDK to setup the NIC port, so DPDK is ready when QEMU launches and connects. Surely as part of that NIC port setup, it could directly tell DPDK which version or featureset to permit on the port ? It is not obvious why the version string has to be fed in via QEMU. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://entangle-photo.org -o-http://search.cpan.org/~danberr/ :| ___ dev mailing list d...@openvswitch.org
Re: [ovs-dev] [libvirt] [RFC] Vhost-user backends cross-version migration support
On 02/01/2017 09:35 AM, Maxime Coquelin wrote: > Hi, > > Few months ago, Michael reported a problem about migrating VMs relying > on vhost-user between hosts supporting different backend versions: > - Message-Id: <20161011173526-mutt-send-email-...@kernel.org> > - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html > > The goal of this thread is to draft a proposal based on the outcomes > of discussions with contributors of the different parties (DPDK/OVS > /libvirt/...). > > Problem statement: > == > > When migrating a VM from one host to another, the interfaces exposed by > QEMU must stay unchanged in order to guarantee a successful migration. > In the case of vhost-user interface, parameters like supported Virtio > feature set, max number of queues, max vring sizes,... must remain > compatible. Indeed, the frontend not being re-initialized, no > renegotiation happens at migration time. > > For example, we have a VM that runs on host A, which has its vhost-user > backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest > also support this feature, it is successfully negotiated, and guest > transmit packets using indirect descriptor tables, that the backend > knows to handle. > At some point, the VM is being migrated to host B, which runs an older > version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC > feature. The migration would break, because the Guest still have the > VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some > decriptors pointing to indirect tables, that backend B doesn't know to > handle. > This is just an example about Virtio features compatibility, but other > backend implementation details could cause other failures. Exactly. Libvirt can't possibly know which virtio features has guest negotiated to use. Therefore I don't think this falls into libvirt scope. > > What we need is to be able to query the destination host's backend to > ensure migration is possible. Also, we would need to query this > statically, even before the VM is started, to be sure it could be > migrated elsewhere for any reason. Again, if you have more than two hosts, say A-Z, I don't see how libvirt could know what hosts to asks (where you will migrate your guest), and what combination of virtio features is okay and which is a deal breaker. > > > Solution 1: Libvirt queries DPDK vhost lib: *KO* > > > Initial idea was to have the management tool (libvirt) to query DPDK > vhost lib and get key/value pairs and check whether migration is > possible. This solution doesn't work for several reasons: > 1. Vhost lib API provide a way for the application to disable features > at runtime (understand, not at build time). So coming back to previous > example, DPDK v16.11 supports indirect descriptor features, but it could > be disabled by OVS. We had a look at whether this API was mandatory, and > it turns out to be, as TSO feature is supported on DPDK but not in OVS. > So we cannot rely on DPDK only. > 2. Some parameter may be not only DPDK specific, such as the maximum > number of queues for example. > > > Solution 2: Libvirt queries OVS for vhost backend key/value pairs: *KO* > === > > Second idea was for OVS to expose its vhost backend implementation > parameters as key/value pairs, for example in the DB or by a dedicated > tool. For example, you could have this kind of information: > - virtio-features: 0x12045694 > - max-rx-queues: 1024 > - max-rx-ring-size: 512 > Doing this, Libvirt has the information to take decision whether > migration is possible or not. > The problem is that Libvirt doesn't know (and want) to interpret these > values (should it be equal/lower/greater/...?), and each time a new > key is introduced in OVS, Libvirt will have to be updated to handle it, > adding an unwanted synchronization constraint between the projects. > > > Solution 3: Libvirt queries OVS for vhost backend version string: *OK* > == > > > The idea is to have a table of supported versions, associated to > key/value pairs. Libvirt could query the list of supported versions > strings for each hosts, and select the first common one among all hosts. How does libvirt know what hosts to ask? Libvirt aims on managing a single host. It has no knowledge of other hosts on the network. That's task for upper layers like RHEV, OpenStack, etc. > > Then, libvirt would ask OVS to probe the vhost-user interfaces in the > selected version (compatibility mode). For example host A runs OVS-2.7, > and host B OVS-2.6. Host A's OVS-2.7 has an OVS-2.6 compatibility mode > (e.g. with indirect descriptors disabled), which should be selected at > vhost-user interface probe time. > > Advantage of doing so is that libvirt does not need any update if new > keys
