Re: Using virtio for inter-VM communication
On 2014-06-17 07:24, Paolo Bonzini wrote: Il 15/06/2014 08:20, Jan Kiszka ha scritto: I think implementing Xen hypercalls in jailhouse for grant table and event channels would actually make a lot of sense. The Xen implementation is 2.5kLOC and I think it should be possible to compact it noticeably, especially if you limit yourself to 64-bit guests. At least the grant table model seems unsuited for Jailhouse. It allows a guest to influence the mapping of another guest during runtime. This we want (or even have) to avoid in Jailhouse. IIRC implementing the grant table hypercalls with copies is inefficient but valid. Back to #1: This is what Rusty is suggesting for virtio. Nothing to win with grant tables then. And if we really have to copy, I would prefer to use a standard. I guess we need to play with prototypes to assess feasibility and impact on existing code. Jan signature.asc Description: OpenPGP digital signature ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
On 2014-06-13 10:45, Paolo Bonzini wrote: Il 13/06/2014 08:23, Jan Kiszka ha scritto: That would preserve zero-copy capabilities (as long as you can work against the shared mem directly, e.g. doing DMA from a physical NIC or storage device into it) and keep the hypervisor out of the loop. This seems ill thought out. How will you program a NIC via the virtio protocol without a hypervisor? And how will you make it safe? You'll need an IOMMU. But if you have an IOMMU you don't need shared memory. Scenarios behind this are things like driver VMs: You pass through the physical hardware to a driver guest that talks to the hardware and relays data via one or more virtual channels to other VMs. This confines a certain set of security and stability risks to the driver VM. I think implementing Xen hypercalls in jailhouse for grant table and event channels would actually make a lot of sense. The Xen implementation is 2.5kLOC and I think it should be possible to compact it noticeably, especially if you limit yourself to 64-bit guests. At least the grant table model seems unsuited for Jailhouse. It allows a guest to influence the mapping of another guest during runtime. This we want (or even have) to avoid in Jailhouse. I'm therefore more in favor of a model where the shared memory region is defined on cell (guest) creation by adding a virtual device that comes with such a region. Jan It should also be almost enough to run Xen PVH guests as jailhouse partitions. If later Xen starts to support virtio, you will get that for free. Paolo signature.asc Description: OpenPGP digital signature ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
Il 15/06/2014 08:20, Jan Kiszka ha scritto: I think implementing Xen hypercalls in jailhouse for grant table and event channels would actually make a lot of sense. The Xen implementation is 2.5kLOC and I think it should be possible to compact it noticeably, especially if you limit yourself to 64-bit guests. At least the grant table model seems unsuited for Jailhouse. It allows a guest to influence the mapping of another guest during runtime. This we want (or even have) to avoid in Jailhouse. IIRC implementing the grant table hypercalls with copies is inefficient but valid. Paolo ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
On 2014-06-13 02:47, Rusty Russell wrote: Jan Kiszka jan.kis...@siemens.com writes: On 2014-06-12 04:27, Rusty Russell wrote: Henning Schild henning.sch...@siemens.com writes: It was also never implemented, and remains a thought experiment. However, implementing it in lguest should be fairly easy. The reason why a trusted helper, i.e. additional logic in the hypervisor, is not our favorite solution is that we'd like to keep the hypervisor as small as possible. I wouldn't exclude such an approach categorically, but we have to weigh the costs (lines of code, additional hypervisor interface) carefully against the gain (existing specifications and guest driver infrastructure). Reasonable, but I think you'll find it is about the minimal implementation in practice. Unfortunately, I don't have time during the next 6 months to implement it myself :( Back to VIRTIO_F_RING_SHMEM_ADDR (which you once brought up in an MCA working group discussion): What speaks against introducing an alternative encoding of addresses inside virtio data structures? The idea of this flag was to replace guest-physical addresses with offsets into a shared memory region associated with or part of a virtio device. We would also need a way of defining the shared memory region. But that's not the problem. If such a feature is not accepted by the guest? How to you fall back? Depends on the hypervisor and its scope, but it should be quite straightforward: full-featured ones like KVM could fall back to slow copying, specialized ones like Jailhouse would clear FEATURES_OK if the guest driver does not accept it (because there would be no ring walking or copying code in Jailhouse), thus refuse the activate the device. That would be absolutely fine for application domains of specialized hypervisors (often embedded, customized guests etc.). The shared memory regions could be exposed as a BARs (PCI) or additional address ranges (device tree) and addressed in the redefined guest address fields via some region index and offset. We don't add features which unmake the standard. That would preserve zero-copy capabilities (as long as you can work against the shared mem directly, e.g. doing DMA from a physical NIC or storage device into it) and keep the hypervisor out of the loop. This seems ill thought out. How will you program a NIC via the virtio protocol without a hypervisor? And how will you make it safe? You'll need an IOMMU. But if you have an IOMMU you don't need shared memory. Scenarios behind this are things like driver VMs: You pass through the physical hardware to a driver guest that talks to the hardware and relays data via one or more virtual channels to other VMs. This confines a certain set of security and stability risks to the driver VM. Is it too invasive to existing infrastructure or does it have some other pitfalls? You'll have to convince every vendor to implement your addition to the standard. Which is easier than inventing a completely new system, but it's not quite virtio. It would be an optional addition, a feature all three sides (host and the communicating guests) would have to agree on. I think we would only have to agree on extending the spec to enable this - after demonstrating it via an implementation, of course. Thanks, Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
Il 13/06/2014 08:23, Jan Kiszka ha scritto: That would preserve zero-copy capabilities (as long as you can work against the shared mem directly, e.g. doing DMA from a physical NIC or storage device into it) and keep the hypervisor out of the loop. This seems ill thought out. How will you program a NIC via the virtio protocol without a hypervisor? And how will you make it safe? You'll need an IOMMU. But if you have an IOMMU you don't need shared memory. Scenarios behind this are things like driver VMs: You pass through the physical hardware to a driver guest that talks to the hardware and relays data via one or more virtual channels to other VMs. This confines a certain set of security and stability risks to the driver VM. I think implementing Xen hypercalls in jailhouse for grant table and event channels would actually make a lot of sense. The Xen implementation is 2.5kLOC and I think it should be possible to compact it noticeably, especially if you limit yourself to 64-bit guests. It should also be almost enough to run Xen PVH guests as jailhouse partitions. If later Xen starts to support virtio, you will get that for free. Paolo ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
Vincent JARDIN vincent.jar...@6wind.com writes: On 10/06/2014 18:48, Henning Schild wrote: Hi, In a first prototype i implemented a ivshmem[2] device for the hypervisor. That way we can share memory between virtual machines. Ivshmem is nice and simple but does not seem to be used anymore. And it does not define higher level devices, like a console. FYI, ivhsmem is used here: http://dpdk.org/browse/memnic/tree/ http://dpdk.org/browse/memnic/tree/pmd/pmd_memnic.c#n449 There are some few other references too, if needed. It may be used, but that doesn't mean it's maintained, or robust against abuse. My advice is to steer clear of it. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
On Thu, 12 Jun 2014 08:48:04 +0200 Markus Armbruster arm...@redhat.com wrote: Vincent JARDIN vincent.jar...@6wind.com writes: On 10/06/2014 18:48, Henning Schild wrote: Hi, In a first prototype i implemented a ivshmem[2] device for the hypervisor. That way we can share memory between virtual machines. Ivshmem is nice and simple but does not seem to be used anymore. And it does not define higher level devices, like a console. FYI, ivhsmem is used here: http://dpdk.org/browse/memnic/tree/ http://dpdk.org/browse/memnic/tree/pmd/pmd_memnic.c#n449 There are some few other references too, if needed. It may be used, but that doesn't mean it's maintained, or robust against abuse. My advice is to steer clear of it. Could you elaborate on why you advice against it? Henning ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
On 12/06/2014 09:44, Henning Schild wrote: It may be used, but that doesn't mean it's maintained, or robust against abuse. My advice is to steer clear of it. Could you elaborate on why you advice against it? +1 elaborate please. beside the DPDK source code, some other common use cases: - HPC: using inter VM shared memory for ultra low latency inter VM computation Best regards, Vincent ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [Qemu-devel] Using virtio for inter-VM communication
Henning Schild henning.sch...@siemens.com writes: On Thu, 12 Jun 2014 08:48:04 +0200 Markus Armbruster arm...@redhat.com wrote: Vincent JARDIN vincent.jar...@6wind.com writes: On 10/06/2014 18:48, Henning Schild wrote: Hi, In a first prototype i implemented a ivshmem[2] device for the hypervisor. That way we can share memory between virtual machines. Ivshmem is nice and simple but does not seem to be used anymore. And it does not define higher level devices, like a console. FYI, ivhsmem is used here: http://dpdk.org/browse/memnic/tree/ http://dpdk.org/browse/memnic/tree/pmd/pmd_memnic.c#n449 There are some few other references too, if needed. It may be used, but that doesn't mean it's maintained, or robust against abuse. My advice is to steer clear of it. Could you elaborate on why you advice against it? ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Why I advise against using ivshmem (was: [Qemu-devel] Using virtio for inter-VM communication)
Henning Schild henning.sch...@siemens.com writes: On Thu, 12 Jun 2014 08:48:04 +0200 Markus Armbruster arm...@redhat.com wrote: Vincent JARDIN vincent.jar...@6wind.com writes: On 10/06/2014 18:48, Henning Schild wrote: Hi, In a first prototype i implemented a ivshmem[2] device for the hypervisor. That way we can share memory between virtual machines. Ivshmem is nice and simple but does not seem to be used anymore. And it does not define higher level devices, like a console. FYI, ivhsmem is used here: http://dpdk.org/browse/memnic/tree/ http://dpdk.org/browse/memnic/tree/pmd/pmd_memnic.c#n449 There are some few other references too, if needed. It may be used, but that doesn't mean it's maintained, or robust against abuse. My advice is to steer clear of it. Could you elaborate on why you advice against it? Sure! The reasons for my dislike range from practical to philosophical. My practical concerns include: 1. ivshmem code needs work, but has no maintainer - Error handling is generally poor. For instance, device_add ivshmem kills your guest instantly. - More subjectively, I don't trust the code to be robust against abuse by our own guest, or the other guests sharing the memory. Convincing me would take a code audit. - MAINTAINERS doesn't cover ivshmem.c. - The last non-trivial commit that isn't obviously part of some tree-wide infrastructure or cleanup work is from September 2012 (commit c08ba66). 2. There is no libvirt support 3. Out-of-tree server program required for full functionality Interrupts require a shared memory server running in the host (see docs/specs/ivshmem_device_spec.txt). It doesn't tell where to find one. The initial commit 6cbf4c8 points to www.gitorious.org/nahanni. That repository's last commit is from September 2012. He's dead, Jim. ivshmem_device_spec.txt is silent on what the server is supposed to do. If this server requires privileges: I don't trust it without an audit. 4. Out-of-tree kernel uio driver required The device is intended to be used with the provided UIO driver (ivshmem_device_spec.txt again). As far as I can tell, the provided UIO driver is the one in the dead Nahanni repo. By now, you should be expecting this: I don't trust that one either. These concerns are all fixable, but it'll take serious work, and time. Something like: * Find a maintainer for the device model * Review and fix its code * Get the required kernel module upstream * Get all the required parts outside QEMU packaged in major distros, or absorbed into QEMU In short, create a viable community around ivshmem, either within the QEMU community, or separately but cooperating. On to the more philosophical ones. 5. Out-of-tree interface required Paraphrasing an old quip: Some people, when confronted with a problem, think I know, I'll use shared memory. Now they have two problems. Shared memory is not an interface. It's at best something you can use to build an interface. I'd rather have us offer something with a little bit more structure. Very fast guest-to-guest networking perhaps. 6. Device models belong into QEMU Say you build an actual interface on top of ivshmem. Then ivshmem in QEMU together with the supporting host code outside QEMU (see 3.) and the lower layer of the code using it in guests (kernel + user space) provide something that to me very much looks like a device model. Device models belong into QEMU. It's what QEMU does. To all currently using ivshmem or contemplating its use: I'd like to invite you to work with the QEMU community to get your use case served better. You could start worse than with explaining it to us. In case you'd rather not work with the QEMU community: I'm not passing judgement on that (heck, I have had days when I'd rather not, too). But if somebody's reasons not to work with us include GPL circumvention, then that somebody is a scoundrel. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
Jan Kiszka jan.kis...@siemens.com writes: On 2014-06-12 04:27, Rusty Russell wrote: Henning Schild henning.sch...@siemens.com writes: It was also never implemented, and remains a thought experiment. However, implementing it in lguest should be fairly easy. The reason why a trusted helper, i.e. additional logic in the hypervisor, is not our favorite solution is that we'd like to keep the hypervisor as small as possible. I wouldn't exclude such an approach categorically, but we have to weigh the costs (lines of code, additional hypervisor interface) carefully against the gain (existing specifications and guest driver infrastructure). Reasonable, but I think you'll find it is about the minimal implementation in practice. Unfortunately, I don't have time during the next 6 months to implement it myself :( Back to VIRTIO_F_RING_SHMEM_ADDR (which you once brought up in an MCA working group discussion): What speaks against introducing an alternative encoding of addresses inside virtio data structures? The idea of this flag was to replace guest-physical addresses with offsets into a shared memory region associated with or part of a virtio device. We would also need a way of defining the shared memory region. But that's not the problem. If such a feature is not accepted by the guest? How to you fall back? We don't add features which unmake the standard. That would preserve zero-copy capabilities (as long as you can work against the shared mem directly, e.g. doing DMA from a physical NIC or storage device into it) and keep the hypervisor out of the loop. This seems ill thought out. How will you program a NIC via the virtio protocol without a hypervisor? And how will you make it safe? You'll need an IOMMU. But if you have an IOMMU you don't need shared memory. Is it too invasive to existing infrastructure or does it have some other pitfalls? You'll have to convince every vendor to implement your addition to the standard. Which is easier than inventing a completely new system, but it's not quite virtio. Cheers, Rusty. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
Henning Schild henning.sch...@siemens.com writes: Hi, i am working on the jailhouse[1] project and am currently looking at inter-VM communication. We want to connect guests directly with virtual consoles based on shared memory. The code complexity in the hypervisor should be minimal, it should just make the shared memory discoverable and provide a signaling mechanism. Hi Henning, The virtio assumption was that the host can see all of guest memory. This simplifies things significantly, and makes it efficient. If you don't have this, *someone* needs to do a copy. Usually the guest OS does a bounce buffer into your shared region. Goodbye performance. Or you can play remapping tricks. Goodbye performance again. My preferred model is to have a trusted helper (ie. host) which understands how to copy between virtio rings. The backend guest (to steal Xen vocab) R/O maps the descriptor, avail ring and used rings in the guest. It then asks the trusted helper to do various operation (copy into writable descriptor, copy out of readable descriptor, mark used). The virtio ring itself acts as a grant table. Note: that helper mechanism is completely protocol agnostic. It was also explicitly designed into the virtio mechanism (with its 4k boundaries for data structures and its 'len' field to indicate how much was written into the descriptor). It was also never implemented, and remains a thought experiment. However, implementing it in lguest should be fairly easy. Cheers, Rusty. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
On 2014-06-12 04:27, Rusty Russell wrote: Henning Schild henning.sch...@siemens.com writes: Hi, i am working on the jailhouse[1] project and am currently looking at inter-VM communication. We want to connect guests directly with virtual consoles based on shared memory. The code complexity in the hypervisor should be minimal, it should just make the shared memory discoverable and provide a signaling mechanism. Hi Henning, The virtio assumption was that the host can see all of guest memory. This simplifies things significantly, and makes it efficient. If you don't have this, *someone* needs to do a copy. Usually the guest OS does a bounce buffer into your shared region. Goodbye performance. Or you can play remapping tricks. Goodbye performance again. My preferred model is to have a trusted helper (ie. host) which understands how to copy between virtio rings. The backend guest (to steal Xen vocab) R/O maps the descriptor, avail ring and used rings in the guest. It then asks the trusted helper to do various operation (copy into writable descriptor, copy out of readable descriptor, mark used). The virtio ring itself acts as a grant table. Note: that helper mechanism is completely protocol agnostic. It was also explicitly designed into the virtio mechanism (with its 4k boundaries for data structures and its 'len' field to indicate how much was written into the descriptor). It was also never implemented, and remains a thought experiment. However, implementing it in lguest should be fairly easy. The reason why a trusted helper, i.e. additional logic in the hypervisor, is not our favorite solution is that we'd like to keep the hypervisor as small as possible. I wouldn't exclude such an approach categorically, but we have to weigh the costs (lines of code, additional hypervisor interface) carefully against the gain (existing specifications and guest driver infrastructure). Back to VIRTIO_F_RING_SHMEM_ADDR (which you once brought up in an MCA working group discussion): What speaks against introducing an alternative encoding of addresses inside virtio data structures? The idea of this flag was to replace guest-physical addresses with offsets into a shared memory region associated with or part of a virtio device. That would preserve zero-copy capabilities (as long as you can work against the shared mem directly, e.g. doing DMA from a physical NIC or storage device into it) and keep the hypervisor out of the loop. Is it too invasive to existing infrastructure or does it have some other pitfalls? Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Using virtio for inter-VM communication
Hi, i am working on the jailhouse[1] project and am currently looking at inter-VM communication. We want to connect guests directly with virtual consoles based on shared memory. The code complexity in the hypervisor should be minimal, it should just make the shared memory discoverable and provide a signaling mechanism. We would like to reuse virtio so that Linux-guests will eventually just work without having to patch them. Having looked at virtio it seems to be focused on host-guest communication and does not consider direct guest-guest communication. I.e. the queues use guest-physical addressing, which is only meaningful for the guest and the host. In a first prototype i implemented a ivshmem[2] device for the hypervisor. That way we can share memory between virtual machines. Ivshmem is nice and simple but does not seem to be used anymore. And it does not define higher level devices, like a console. At this point i could: - define a console on top of ivshmem - see how i can get a virtio console to work between guests on shared memory Is anyone already using something like that? I guess zero-copy virtio devices in Xen would be a similar case. I read a suggestion from may 2010 to introduce a virtio feature bit for shared memory (VIRTIO_F_RING_SHMEM_ADDR). But that did not make it into the virtio-spec. regards, Henning [1] jailhouse https://github.com/siemens/jailhouse [2] ivshmem https://gitorious.org/nahanni ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: Using virtio for inter-VM communication
On 10/06/2014 18:48, Henning Schild wrote: Hi, In a first prototype i implemented a ivshmem[2] device for the hypervisor. That way we can share memory between virtual machines. Ivshmem is nice and simple but does not seem to be used anymore. And it does not define higher level devices, like a console. FYI, ivhsmem is used here: http://dpdk.org/browse/memnic/tree/ http://dpdk.org/browse/memnic/tree/pmd/pmd_memnic.c#n449 There are some few other references too, if needed. Best regards, Vincent ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
