Re: [RFC 0/4] Virtio uses DMA API for all devices
On Sun, 2018-08-05 at 03:22 +0300, Michael S. Tsirkin wrote: > I see the allure of this, but I think down the road you will > discover passing a flag in libvirt XML saying > "please use a secure mode" or whatever is a good idea. > > Even thought it is probably not required to address this > specific issue. > > For example, I don't think ballooning works in secure mode, > you will be able to teach libvirt not to try to add a > balloon to the guest. Right, we'll need some quirk to disable balloons in the guest I suppose. Passing something from libvirt is cumbersome because the end user may not even need to know about secure VMs. There are use cases where the security is a contract down to some special application running inside the secure VM, the sysadmin knows nothing about. Also there's repercussions all the way to admin tools, web UIs etc... so it's fairly wide ranging. So as long as we only need to quirk a couple of devices, it's much better contained that way. > > Later on, (we may have even already run Linux at that point, > > unsecurely, as we can use Linux as a bootloader under some > > circumstances), we start a "secure image". > > > > This is a kernel zImage that includes a "ticket" that has the > > appropriate signature etc... so that when that kernel starts, it can > > authenticate with the ultravisor, be verified (along with its ramdisk) > > etc... and copied (by the UV) into secure memory & run from there. > > > > At that point, the hypervisor is informed that the VM has become > > secure. > > > > So at that point, we could exit to qemu to inform it of the change, > > That's probably a good idea too. We probably will have to tell qemu eventually for migration, as we'll need some kind of key exchange phase etc... to deal with the crypto aspects (the actual page copy is sorted via encrypting the secure pages back to normal pages in qemu, but we'll need extra metadata). > > and > > have it walk the qtree and "Switch" all the virtio devices to use the > > IOMMU I suppose, but it feels a lot grosser to me. > > That part feels gross, yes. > > > That's the only other option I can think of. > > > > > However in this specific case, the flag does not need to come from the > > > hypervisor, it can be set by arch boot code I think. > > > Christoph do you see a problem with that? > > > > The above could do that yes. Another approach would be to do it from a > > small virtio "quirk" that pokes a bit in the device to force it to > > iommu mode when it detects that we are running in a secure VM. That's a > > bit warty on the virito side but probably not as much as having a qemu > > one that walks of the virtio devices to change how they behave. > > > > What do you reckon ? > > I think you are right that for the dma limit the hypervisor doesn't seem > to need to know. It's not just a limit mind you. It's a range, at least if we allocate just a single pool of insecure pages. swiotlb feels like a better option for us. > > What we want to avoid is to expose any of this to the *end user* or > > libvirt or any other higher level of the management stack. We really > > want that stuff to remain contained between the VM itself, KVM and > > maybe qemu. > > > > We will need some other qemu changes for migration so that's ok. But > > the minute you start touching libvirt and the higher levels it becomes > > a nightmare. > > > > Cheers, > > Ben. > > I don't believe you'll be able to avoid that entirely. The split between > libvirt and qemu is more about community than about code, random bits of > functionality tend to land on random sides of that fence. Better add a > tag in domain XML early is my advice. Having said that, it's your > hypervisor. I'm just suggesting that when hypervisor does somehow need > to care then I suspect most people won't be receptive to the argument > that changing libvirt is a nightmare. It only needs to care at runtime. The problem isn't changing libvirt per-se, I don't have a problem with that. The problem is that it means creating two categories of machines "secure" and "non-secure", which is end-user visible, and thus has to be escalated to all the various management stacks, UIs, etc... out there. In addition, there are some cases where the individual creating the VMs may not have any idea that they are secure. But yes, if we have to, we'll do it. However, so far, we don't think it's a great idea. Cheers, Ben. > > > > > To get swiotlb you'll need to then use the DT/ACPI > > > > > dma-range property to limit the addressable range, and a swiotlb > > > > > capable plaform will use swiotlb automatically. > > > > > > > > This cannot be done as you describe it. > > > > > > > > The VM is created as a *normal* VM. The DT stuff is generated by qemu > > > > at a point where it has *no idea* that the VM will later become secure > > > > and thus will have to restrict which pages can be used for "DMA". > > > > > > > > The VM will *at runtime* turn itself into a secure
Re: [RFC 0/4] Virtio uses DMA API for all devices
On Sat, 2018-08-04 at 01:21 -0700, Christoph Hellwig wrote: > No matter if you like it or not (I don't!) virtio is defined to bypass > dma translations, it is very clearly stated in the spec. It has some > ill-defined bits to bypass it, so if you want the dma mapping API > to be used you'll have to set that bit (in its original form, a refined > form, or an entirely newly defined sane form) and make sure your > hypersivors always sets it. It's not rocket science, just a little bit > for work to make sure your setup is actually going to work reliably > and portably. I think you are conflating completely different things, let me try to clarify, we might actually be talking past each other. > > We aren't going to cancel years of HW and SW development for our > > Maybe you should have actually read the specs you are claiming to > implemented before spending all that effort. Anyway, let's cool our respective jets and sort that out, there are indeed other approaches than overriding the DMA ops with special ones, though I find them less tasty ... but here's my attempt at a (simpler) description. Bear with me for the long-ish email, this tries to describe the system so you get an idea where we come from, and options we can use to get out of this. So we *are* implementing the spec, since qemu is currently unmodified: Default virtio will bypass the iommu emulated by qemu as per spec etc.. On the Linux side, thus, virtio "sees" a normal iommu-bypassing device and will treat it as such. The problem is the assumption in the middle that qemu can access all guest pages directly, which holds true for traditional VMs, but breaks when the VM in our case turns itself into a secure VM. This isn't under the action (or due to changes in) the hypervisor. KVM operates (almost) normally here. But there's this (very thin and open source btw) layer underneath called ultravisor, which exploits some HW facilities to maintain a separate pool of "secure" memory, which cannot be physically accessed by a non-secure entity. So in our scenario, qemu and KVM create a VM totally normally, there is no changes required to the VM firmware, bootloader(s), etc... in fact we support Linux based bootloaders, and those will work as normal linux would in a VM, virtio works normally, etc... Until that VM (via grub or kexec for example) loads a "secure image". That secure image is a Linux kernel which has been "wrapped" (to simply imagine a modified zImage wrapper though that's not entirely exact). When that is run, before it modifies it's .data, it will interact with the ultravisor using a specific HW facility to make itself secure. What happens then is that the UV cryptographically verifies the kernel and ramdisk, and copies them to the secure memory where execution returns. The Ultravisor is then involved as a small shim for hypercalls between the secure VM and KVM to prevent leakage of information (sanitize registers etc...). Now at this point, qemu can no longer access the secure VM pages (there's more to this, such as using HMM to allow migration/encryption accross etc... but let's not get bogged down). So virtio can no longer access any page in the VM. Now the VM *can* request from the Ultravisor some selected pages to be made "insecure" and thus shared with qemu. This is how we handle some of the pages used in our paravirt stuff, and that's how we want to deal with virtio, by creating an insecure swiotlb pool. At this point, thus, there are two options. - One you have rejected, which is to have a way for "no-iommu" virtio (which still doesn't use an iommu on the qemu side and doesn't need to), to be forced to use some custom DMA ops on the VM side. - One, which sadly has more overhead and will require modifying more pieces of the puzzle, which is to make qemu uses an emulated iommu. Once we make qemu do that, we can then layer swiotlb on top of the emulated iommu on the guest side, and pass that as dma_ops to virtio. Now, assuming you still absolutely want us to go down the second option, there are several ways to get there. We would prefer to avoid requiring the user to pass some special option to qemu. That has an impact up the food chain (libvirt, management tools etc...) and users probably won't understand what it's about. In fact the *end user* might not even need to know a VM is secure, though applications inside might. There's the additional annoyance that currently our guest FW (SLOF) cannot deal with virtio in IOMMU mode, but that's fixable. >From there, refer to the email chain between Michael and I where we are discussing options to "switch" virtio at runtime on the qemu side. Any comment or suggestion ? Cheers, Ben. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC 0/4] Virtio uses DMA API for all devices
On Sun, 2018-08-05 at 03:09 +0300, Michael S. Tsirkin wrote: > It seems that the fact that within guest it's implemented using a bounce > buffer and that it's easiest to do by switching virtio to use the DMA API > isn't something virtio spec concerns itself with. Right, this is my reasoning as well. See this other (long) email I just sent to Christoph to explain the whole flow. > I'm open to suggestions. Cheers, Ben. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC 0/4] Virtio uses DMA API for all devices
On Wed, Aug 01, 2018 at 09:16:38AM +0100, Will Deacon wrote: > On Tue, Jul 31, 2018 at 03:36:22PM -0500, Benjamin Herrenschmidt wrote: > > On Tue, 2018-07-31 at 10:30 -0700, Christoph Hellwig wrote: > > > > However the question people raise is that DMA API is already full of > > > > arch-specific tricks the likes of which are outlined in your post linked > > > > above. How is this one much worse? > > > > > > None of these warts is visible to the driver, they are all handled in > > > the architecture (possibly on a per-bus basis). > > > > > > So for virtio we really need to decide if it has one set of behavior > > > as specified in the virtio spec, or if it behaves exactly as if it > > > was on a PCI bus, or in fact probably both as you lined up. But no > > > magic arch specific behavior inbetween. > > > > The only arch specific behaviour is needed in the case where it doesn't > > behave like PCI. In this case, the PCI DMA ops are not suitable, but in > > our secure VMs, we still need to make it use swiotlb in order to bounce > > through non-secure pages. > > On arm/arm64, the problem we have is that legacy virtio devices on the MMIO > transport (so definitely not PCI) have historically been advertised by qemu > as not being cache coherent, but because the virtio core has bypassed DMA > ops then everything has happened to work. If we blindly enable the arch DMA > ops, we'll plumb in the non-coherent ops and start getting data corruption, > so we do need a way to quirk virtio as being "always coherent" if we want to > use the DMA ops (which we do, because our emulation platforms have an IOMMU > for all virtio devices). > > Will Right that's not very different from placing the device within the IOMMU domain but in fact bypassing the IOMMU. I wonder whether anyone ever needs a non coherent virtio-mmio. If yes we can extend PLATFORM_IOMMU to cover that or add another bit. What exactly do the non-coherent ops do that causes the corruption? -- MST ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC 0/4] Virtio uses DMA API for all devices
On Fri, Aug 03, 2018 at 08:22:11PM -0500, Benjamin Herrenschmidt wrote: > (Appologies if you got this twice, my mailer had a brain fart and I don't > know if the first one got through & am about to disappear in a plane for 17h) I got like 3 of these. I hope that's true for everyone as I replied to 1st one. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC 0/4] Virtio uses DMA API for all devices
On Fri, Aug 03, 2018 at 08:21:26PM -0500, Benjamin Herrenschmidt wrote: > On Fri, 2018-08-03 at 22:08 +0300, Michael S. Tsirkin wrote: > > > > > Please go through these patches and review whether this approach > > > > > broadly > > > > > makes sense. I will appreciate suggestions, inputs, comments regarding > > > > > the patches or the approach in general. Thank you. > > > > > > > > Jason did some work on profiling this. Unfortunately he reports > > > > about 4% extra overhead from this switch on x86 with no vIOMMU. > > > > > > The test is rather simple, just run pktgen (pktgen_sample01_simple.sh) in > > > guest and measure PPS on tap on host. > > > > > > Thanks > > > > Could you supply host configuration involved please? > > I wonder how much of that could be caused by Spectre mitigations > blowing up indirect function calls... > > Cheers, > Ben. I won't be surprised. If yes I suggested a way to mitigate the overhead. -- MSR ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC 0/4] Virtio uses DMA API for all devices
On Fri, Aug 03, 2018 at 08:16:21PM -0500, Benjamin Herrenschmidt wrote: > On Fri, 2018-08-03 at 22:07 +0300, Michael S. Tsirkin wrote: > > On Fri, Aug 03, 2018 at 10:58:36AM -0500, Benjamin Herrenschmidt wrote: > > > On Fri, 2018-08-03 at 00:05 -0700, Christoph Hellwig wrote: > > > > > 2- Make virtio use the DMA API with our custom platform-provided > > > > > swiotlb callbacks when needed, that is when not using IOMMU *and* > > > > > running on a secure VM in our case. > > > > > > > > And total NAK the customer platform-provided part of this. We need > > > > a flag passed in from the hypervisor that the device needs all bus > > > > specific dma api treatment, and then just use the normal plaform > > > > dma mapping setup. > > > > > > Christoph, as I have explained already, we do NOT have a way to provide > > > such a flag as neither the hypervisor nor qemu knows anything about > > > this when the VM is created. > > > > I think the fact you can't add flags from the hypervisor is > > a sign of a problematic architecture, you should look at > > adding that down the road - you will likely need it at some point. > > Well, we can later in the boot process. At VM creation time, it's just > a normal VM. The VM firmware, bootloader etc... are just operating > normally etc... I see the allure of this, but I think down the road you will discover passing a flag in libvirt XML saying "please use a secure mode" or whatever is a good idea. Even thought it is probably not required to address this specific issue. For example, I don't think ballooning works in secure mode, you will be able to teach libvirt not to try to add a balloon to the guest. > Later on, (we may have even already run Linux at that point, > unsecurely, as we can use Linux as a bootloader under some > circumstances), we start a "secure image". > > This is a kernel zImage that includes a "ticket" that has the > appropriate signature etc... so that when that kernel starts, it can > authenticate with the ultravisor, be verified (along with its ramdisk) > etc... and copied (by the UV) into secure memory & run from there. > > At that point, the hypervisor is informed that the VM has become > secure. > > So at that point, we could exit to qemu to inform it of the change, That's probably a good idea too. > and > have it walk the qtree and "Switch" all the virtio devices to use the > IOMMU I suppose, but it feels a lot grosser to me. That part feels gross, yes. > That's the only other option I can think of. > > > However in this specific case, the flag does not need to come from the > > hypervisor, it can be set by arch boot code I think. > > Christoph do you see a problem with that? > > The above could do that yes. Another approach would be to do it from a > small virtio "quirk" that pokes a bit in the device to force it to > iommu mode when it detects that we are running in a secure VM. That's a > bit warty on the virito side but probably not as much as having a qemu > one that walks of the virtio devices to change how they behave. > > What do you reckon ? I think you are right that for the dma limit the hypervisor doesn't seem to need to know. > What we want to avoid is to expose any of this to the *end user* or > libvirt or any other higher level of the management stack. We really > want that stuff to remain contained between the VM itself, KVM and > maybe qemu. > > We will need some other qemu changes for migration so that's ok. But > the minute you start touching libvirt and the higher levels it becomes > a nightmare. > > Cheers, > Ben. I don't believe you'll be able to avoid that entirely. The split between libvirt and qemu is more about community than about code, random bits of functionality tend to land on random sides of that fence. Better add a tag in domain XML early is my advice. Having said that, it's your hypervisor. I'm just suggesting that when hypervisor does somehow need to care then I suspect most people won't be receptive to the argument that changing libvirt is a nightmare. > > > > To get swiotlb you'll need to then use the DT/ACPI > > > > dma-range property to limit the addressable range, and a swiotlb > > > > capable plaform will use swiotlb automatically. > > > > > > This cannot be done as you describe it. > > > > > > The VM is created as a *normal* VM. The DT stuff is generated by qemu > > > at a point where it has *no idea* that the VM will later become secure > > > and thus will have to restrict which pages can be used for "DMA". > > > > > > The VM will *at runtime* turn itself into a secure VM via interactions > > > with the security HW and the Ultravisor layer (which sits below the > > > HV). This happens way after the DT has been created and consumed, the > > > qemu devices instanciated etc... > > > > > > Only the guest kernel knows because it initates the transition. When > > > that happens, the virtio devices have already been used by the guest > > > firmware, bootloader, possibly another kernel
Re: [RFC 0/4] Virtio uses DMA API for all devices
On Sat, Aug 04, 2018 at 01:15:00AM -0700, Christoph Hellwig wrote: > On Fri, Aug 03, 2018 at 10:17:32PM +0300, Michael S. Tsirkin wrote: > > It seems reasonable to teach a platform to override dma-range > > for a specific device e.g. in case it knows about bugs in ACPI. > > A platform will be able override dma-range using the dev->bus_dma_mask > field starting in 4.19. But we'll still need a way how to > > a) document in the virtio spec that all bus dma quirks are to be > applied I agree it's a good idea. In particular I suspect that PLATFORM_IOMMU should be extended to cover that. But see below. > b) a way to document in a virtio-related spec how the bus handles > dma for Ben's totally fucked up hypervisor. Without that there > is not way we'll get interoperable implementations. So in this case however I'm not sure what exactly do we want to add. It seems that from point of view of the device, there is nothing special - it just gets a PA and writes there. It also seems that guest does not need to get any info from the device either. Instead guest itself needs device to DMA into specific addresses, for its own reasons. It seems that the fact that within guest it's implemented using a bounce buffer and that it's easiest to do by switching virtio to use the DMA API isn't something virtio spec concerns itself with. I'm open to suggestions. -- MST ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH net-next] vhost: switch to use new message format
From: Jason Wang
Date: Fri, 3 Aug 2018 15:04:51 +0800
> So fixing this by introducing a new message type with an explicit
> 32bit reserved field after type like:
>
> struct vhost_msg_v2 {
> int type;
> __u32 reserved;
Please use fixed sized types consistently. Use 's32' instead of 'int'
here.
Thanks!
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC 0/4] Virtio uses DMA API for all devices
On Fri, Aug 03, 2018 at 01:58:46PM -0500, Benjamin Herrenschmidt wrote: > You are saying something along the lines of "I don't like an > instruction in your ISA, let's not support your entire CPU architecture > in Linux". No. I'm saying if you can't describe your architecture in the virtio spec document it is bogus. > Our setup is not fucked. It makes a LOT of sense and it's a very > sensible design. It's hitting a problem due to a corner case oddity in > virtio bypassing the MMU, we've worked around such corner cases many > times in the past without any problem, I fail to see what the problem > is here. No matter if you like it or not (I don't!) virtio is defined to bypass dma translations, it is very clearly stated in the spec. It has some ill-defined bits to bypass it, so if you want the dma mapping API to be used you'll have to set that bit (in its original form, a refined form, or an entirely newly defined sane form) and make sure your hypersivors always sets it. It's not rocket science, just a little bit for work to make sure your setup is actually going to work reliably and portably. > We aren't going to cancel years of HW and SW development for our Maybe you should have actually read the specs you are claiming to implemented before spending all that effort. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC 0/4] Virtio uses DMA API for all devices
On Fri, Aug 03, 2018 at 10:17:32PM +0300, Michael S. Tsirkin wrote: > It seems reasonable to teach a platform to override dma-range > for a specific device e.g. in case it knows about bugs in ACPI. A platform will be able override dma-range using the dev->bus_dma_mask field starting in 4.19. But we'll still need a way how to a) document in the virtio spec that all bus dma quirks are to be applied b) a way to document in a virtio-related spec how the bus handles dma for Ben's totally fucked up hypervisor. Without that there is not way we'll get interoperable implementations. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
