On Tue, Jul 18, 2017 at 03:42:08PM +0200, Maxime Coquelin wrote:
> This is an revival from a thread I initiated earlier this year [0], that
> I had to postpone due to other priorities.
> First, I'd like to thanks reviewers of my first proposal, this new
> version tries to address the comments made:
>  1.This is Nova's role and not Libvirt's to query hosts supported
> compatibility modes and to select one, since Nova adds the vhost-user
> ports and has visibility on other hosts. Hence I remove libvirt ML and
> add Openstack one in the recipient list.
>  2. By default, the compatibility version selected is the most recent
> one, except if the admin selects on older compat version.
> The goal of this thread is to draft a solution based on the outcomes
> of discussions with contributors of the different parties (DPDK/OVS
> /Nova/...).
> I'm really interested on feedback from OVS & Nova contributors,
> as my experience with these projects is rather limited.
> Problem statement:
> ==================
> When migrating a VM from one host to another, the interfaces exposed by
> QEMU must stay unchanged in order to guarantee a successful migration.
> In the case of vhost-user interface, parameters like supported Virtio
> feature set, max number of queues, max vring sizes,... must remain
> compatible. Indeed, the frontend not being re-initialized, no
> re-negotiation happens at migration time.
> For example, we have a VM that runs on host A, which has its vhost-user
> backend advertising VIRTIO_F_RING_INDIRECT_DESC feature. Since the Guest
> also support this feature, it is successfully negotiated, and guest
> transmit packets using indirect descriptor tables, that the backend
> knows to handle.
> At some point, the VM is being migrated to host B, which runs an older
> version of the backend not supporting this VIRTIO_F_RING_INDIRECT_DESC
> feature. The migration would break, because the Guest still have the
> VIRTIO_F_RING_INDIRECT_DESC bit sets, and the virtqueue contains some
> decriptors pointing to indirect tables, that backend B doesn't know to
> handle.
> This is just an example about Virtio features compatibility, but other
> backend implementation details could cause other failures. (e.g.
> configurable queues sizes)
> What we need is to be able to query the destination host's backend to
> ensure migration is possible before it is initiated.

This remided me strongly of the issues around the virtual CPU modeling
in KVM, see

QEMU recently gained query-cpu-model-expansion to allow capability queries.

Cc Eduardo accordingly. Eduardo, could you please take a look -
how is the problem solved on the KVM/VCPU side? Do the above
problem and solution for vhost look similar?

> The below proposal has been drafted based on how Qemu manages machine types:
> Proposal
> ========
> The idea is to have a table of supported version strings in OVS,
> associated to key/value pairs. Nova or any other management tool could
> query OVS for the list of supported versions strings for each hosts.
> By default, the latest compatibility version will be selected, but the
> admin can select manually an older compatibility mode in order to ensure
> successful migration to an older destination host.
> Then, Nova would add OVS's vhost-user port with adding the selected
> version (compatibility mode) as an extra parameter.
> Before starting the VM migration, Nova will ensure both source and
> destination hosts' vhost-user interfaces run in the same compatibility
> modes, and will prevent it if this is not the case.
> For example host A runs OVS-2.7, and host B OVS-2.6.
> Host A's OVS-2.7 has an OVS-2.6 compatibility mode (e.g. with indirect
> descriptors disabled), which should be selected at vhost-user port add
> time to ensure migration will succeed to host B.
> Advantage of doing so is that Nova does not need any update if new keys
> are introduced (i.e. it does not need to know how the new keys have to
> be handled), all these checks remain in OVS's vhost-user implementation.
> Ideally, we would support per vhost-user interface compatibility mode,
> which may have an impact also on DPDK API, as the Virtio feature update
> API is global, and not per port.
> - Implementation:
> -----------------
> Goal here is just to illustrate this proposal, I'm sure you will have
> good suggestion to improve it.
> In OVS vhost-user library, we would introduce a new structure, for
> example (neither compiled nor tested):
> struct vhostuser_compat {
>   char *version;
>   uint64_t virtio_features;
>   uint32_t max_rx_queue_sz;
>   uint32_t max_nr_queues;
> };
> *version* field is the compatibility version string. It could be
> something like: "upstream.ovs-dpdk.v2.6". In case for example Fedora
> adds some more patches to its package that would break migration to
> upstream version, it could have a dedicated compatibility string:
> "fc26.ovs-dpdk.v2.6". In case OVS-v2.7 does not break compatibility with
> previous OVS-v2.6 version, then no need to create a new entry, just keep
> v2.6 one.
> *virtio_features* field is the Virtio features set for a given
> compatibility version. When an OVS tag is to be created, it would be
> associated to a DPDK version. The Virtio features for these version
> would be stored in this field. It would allow to upgrade the DPDK
> package for example from v16.07 to v16.11 without breaking migration.
> In case the distribution wants to benefit from latests Virtio
> features, it would have to create a new entry to ensure migration
> won't be broken.
> *max_rx_queue_sz*
> *max_nr_queues* fields are just here for example, don't think this is
> needed today. I just want to illustrate that we have to anticipate
> other parameters than the Virtio feature set, even if not necessary
> at the moment.
> We create a table with different compatibility versions in OVS
> vhost-user lib:
> static struct vhostuser_compat vu_compat[] = {
>   {
>     .version = "upstream.ovs-dpdk.v2.7",
>     .virtio_features = 0x12045694,
>     .max_rx_queue_sz = 512,
>   },
>   {
>     .version = "upstream.ovs-dpdk.v2.6",
>     .virtio_features = 0x10045694,
>     .max_rx_queue_sz = 1024,
>   },
> };
> At some time during installation, or system init, the table would be
> parsed, and compatibility version strings would be stored into the OVS
> database, or a new tool would be created to list these strings, or a
> config file packaged with OVS stores the list of compatibiliy versions.
> Before launching the VM, Nova will query the version strings for the
> host so that the admin can select an older compatibility mode. If none
> selected by the admin, then the most recent one will be used by default,
> and passed to the OVS's add-port command as parameter. Note that if no
> compatibility mode is passed to the add-port command, the most recent
> one is selected by OVS as default.
> When the vhost-user connection is initiated, OVS would know in which
> compatibility mode to init the interface, for example by restricting the
> support Virtio features of the interface.
> Cheers,
> Maxime
> [0]:
>  https://mail.openvswitch.org/pipermail/ovs-dev/2017-February/328257.html
>  <b2a5501c-7df7-ad2a-002f-d731c445a...@redhat.com>
dev mailing list

Reply via email to