Re: VirtIO MMIO for amd64
Here's the full patch for GENPVH+cmdline MMIO+Firecracker support https://imil.net/NetBSD/genpvh+mmio.patch This patch includes: - generic PVH boot for qemu, Firecracker etc... see the "PVH boot with qemu" thread for details - a new bus, pvbus, used to attach devices that could be used in more than one hypervisor, such as the cmdline MMIO device or pvclock (not ported yet) - MMIO cmdline support, used by qemu and FC to advertise virtio devices memory mappings to guests - a "fix" for MP tables, or like Colin says "bug for bug compatibility", needed to detect IOAPIC on guests with no ACPI - a fix for ld_virtio block device enabling FC to use it The patch applies on the latest current. kernel configuration: pv* at pvbus? virtio* at pv? ld* at virtio? vioif* at virtio? viornd* at virtio? This is how fast this thing boots: https://twitter.com/iMilnb/status/1743576957046931647 Feedback appreciated. Cheers, -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
On 1/6/24 08:33, Emile 'iMil' Heitor wrote: To support this, I've attached `virtio*` to a newly created bus, `cmdlinebus?` which is attached to mainbus in amd64_mainbus.c I am not entirely sure this is the cleanest way to proceed and would like to have your feedback. I like the design of OpenBSD's pvbus, it would be well fitted for this and more https://github.com/openbsd/src/tree/master/sys/dev/pv Thoughts? -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
Update My NetBSD/amd64 branch can now boot using PVH on qemu/x86_64, qemu/microvm and Firecracker using MMIO backed block devices. It also supports multiple VirtIO devices such as NICs. The last blocking point was that Firecracker only supports one data segment and as such needs DMA bouncing which was not enabled for that case (thanks Jason for the hint!). Now before opening a PR I'd like your advice on the "cmdline bus" part. As I explained earlier in this thread, without PCI or ACPI, MMIO devices are "detected" through a kernel parameter, for example: root=ld0a console=com rw -v virtio_mmio.device=512@0xfeb00e00:12 virtio_mmio.device=512@0xfeb00c00:11 To support this, I've attached `virtio*` to a newly created bus, `cmdlinebus?` which is attached to mainbus in amd64_mainbus.c I am not entirely sure this is the cleanest way to proceed and would like to have your feedback. The working branch is here: https://github.com/NetBSDfr/NetBSD-src/tree/mmio_cmdline Cheers, -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
Update I've got current/amd64 booting on an MMIO-backed block device with qemu's microvm machine, in both ACPI and command-line-hack (which passes the "device" address as a kernel parameter). If anybody wants to give it a try: $ qemu-system-x86_64 -M microvm,rtc=on -enable-kvm -m 256 -cpu host -kernel netbsd.gdb -append "root=ld0a console=com rw" -serial stdio -display none -device virtio-blk-device,drive=hd0 -drive file=netbsd.img,format=raw,id=hd0 -no-acpi Note the "rtc=on", without this init_main.c/inittodr(rootfstime) hangs. On the other hand, Firecracker loops on an init sig11 crash, I suspect that might also be related to RTC, I guess I'll have to port a virtio RTC. The branch with this feature is here https://github.com/NetBSDfr/NetBSD-src/tree/mmio_cmdline It includes the GENPVH mode to boot the NetBSD kernel from qemu's -kernel flag. You'll need at least those two: virtio* at cmdlinebus? #virtio* at acpi? ld* at virtio? # Virtio disk device As previously said, virtio* at acpi? also works. -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
On 12/26/23 11:35, Emile 'iMil' Heitor wrote: intercepted (or so it seems); I thought the hang, which actually happens sys/kern/subr_disk.c:disk_read_sectors/biowait, Found it. Wrong bus_space_write_4() in virtio_mmio_setup_queue(), 100% my fault, sorry for the noise. IRQ is triggered, block device is recognized, we're close :) -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
Following up I ported the "Linux MPtable bug"-feature found by Colin and IOAPIC is now detected https://github.com/NetBSD/src/commit/37e57b3a0f464246611a005a350e01e25e091bfe Nevertheless, the disk IRQ from Firecracker is still not intercepted (or so it seems); I thought the hang, which actually happens sys/kern/subr_disk.c:disk_read_sectors/biowait, might be related to bad/missing VirtIO device features but I checked on FreeBSD and the features are correctly computed. Ideas are welcome. -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
On 12/20/23 16:56, Taylor R Campbell wrote:
My guess is that this is not going to be MSI -- it'll either be i8259
or ioapic. In that case, it may work to do this:
Thanks a lot for that, here's the result, unfortunately it still
hangs at the first VOP_OPEN() from subr_disk_open.c and the
handler function never gets triggered:
static int
virtio_mmio_cmdline_alloc_interrupts(struct virtio_mmio_softc *msc)
{
struct virtio_mmio_cmdline_softc *const sc =
(struct virtio_mmio_cmdline_softc *)msc;
struct virtio_softc *const vsc = >sc_sc;
struct ioapic_softc *ioapic;
struct pic *pic;
int irq = sc->margs.irq;
int pin = irq;
ioapic = ioapic_find_bybase(irq);
/* we don't enter here */
if (ioapic != NULL) {
pic = >sc_pic;
pin = irq - pic->pic_vecbase;
irq = -1;
} else
pic = _pic;
msc->sc_ih = intr_establish_xname(irq, pic, pin, IST_EDGE, IPL_BIO,
virtio_mmio_intr, msc, false, device_xname(vsc->sc_dev));
if (msc->sc_ih == NULL) {
aprint_error_dev(vsc->sc_dev,
"failed to establish interrupt\n");
return -1;
}
aprint_normal_dev(vsc->sc_dev, "interrupting on %d\n", irq);
return 0;
}
/* --- */
intr_establish_xname() does get through :
[ 1.030] viommio: 4K@0xd000:5
[ 1.030] virtio0: block device (id 2, rev. 0x02)
[ 1.030] ld0 at virtio0: features: 0x1
[ 1.030] virtio0: interrupting on 5 <---
[ 1.030] ld0: 30720 KB, 60 cyl, 16 head, 63 sec, 512 bytes/sect x
61441 sectors
[ 1.0107398] boot device: ld0
--
Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
> Date: Wed, 20 Dec 2023 08:37:32 +0100
> From: Emile 'iMil' Heitor
>
> On 12/20/23 06:55, Emile 'iMil' Heitor wrote:
>
> > Well that's the thing, I can't find where does MMIO attaches on FreeBSD,
> > they have a very simple way of creating the resources:
>
> After a bit of digging, their virtio_mmio device attaches to "nexus0",
> which if I understand correctly, is our mainbus equivalent.
This is presumably some platform-specific irq numbering scheme.
I skimmed through the FreeBSD sources and it looks like, on x86:
1. irqs [0,16) are probably verbatim i8259 vectors, which you can get
at with intr_establish_xname(irq, _pic, ...)
2. irqs in [0,n), where n is the number of pins, might also be ioapic
vectors (which might overlap with i8259?)
3. mptable might determine more ioapic irq numbers allocated
consecutively after the first zero-based ioapic
4. ACPI MADT, if present, might determine more ioapic irq numbers
starting from GlobalIrqBase
5. irq numbers starting at first_msi_irq, which is one higher than all
the previously allocated irq numbers, are for MSI
It would be nice if someone laid out a map of how these interrupt
numbering schemes work and are related (and how they are numbered on
both FreeBSD and NetBSD). I'm sure I can spend hours poring through
the Intel and ACPI manuals to create the reverse map but it's a pain!
My guess is that this is not going to be MSI -- it'll either be i8259
or ioapic. In that case, it may work to do this:
struct ioapic_soft *ioapic = ioapic_find_bybase(irq);
struct pic *pic;
int pin;
if (ioapic != NULL) {
/*
* Some logic in x86/acpi/acpi_machdep.c has a
* conditional on this, but I can't see how we
* could possibly have a struct ioapic_softc
* whose sc_pic.pic_type is not PIC_IOAPIC, so
* I think that must be a dead branch.
*/
KASSERT(ioapic->sc_pic.pic_type == PIC_IOAPIC);
pic = >sc_pic;
pin = irq - pic->pic_vecbase;
irq = -1;
} else {
pic = _pic;
pin = irq;
}
ih = intr_establish_xname(irq, pic, pin, ...);
Re: VirtIO MMIO for amd64
On 12/20/23 06:55, Emile 'iMil' Heitor wrote: Well that's the thing, I can't find where does MMIO attaches on FreeBSD, they have a very simple way of creating the resources: After a bit of digging, their virtio_mmio device attaches to "nexus0", which if I understand correctly, is our mainbus equivalent. -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
On 12/19/23 13:11, Taylor R Campbell wrote: to queue a task and then run it. Normally softint_schedule is called from a hard interrupt handler. Here, you need something in the host to get hard interrupts. For example, on ACPI systems there are ACPI interrupt resources that can be used with acpi_intr_establish; on FDT systems, device nodes have interrupt properties that can be used with fdtbus_intr_establish. How does FreeBSD get x86 mmio intrs on these systems? Well that's the thing, I can't find where does MMIO attaches on FreeBSD, they have a very simple way of creating the resources: https://github.com/freebsd/freebsd-src/blob/main/sys/dev/virtio/mmio/virtio_mmio_cmdline.c#L109 and even Colin doesn't know where this goes https://x.com/cperciva/status/173720959840762?s=20 What I can say: it's not ACPI nor PCI as they are both unsupported on Firecracker. -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
> Date: Tue, 19 Dec 2023 08:55:32 +0100 > From: Emile 'iMil' Heitor > > Maybe related: the interrupt handler function I wrote uses > softint_establish() as there's no "real" hardware behind this > block device, is this the correct way to deal with it? You need to use softint_schedule in order for the softint to run. Otherwise it doesn't do anything. It doesn't matter whether there's any physical manifestation of the hardware in any philosophical sense of the real world; what matters is how the host triggers some kind of entry into the guest. softint(9) doesn't know anything about hardware/host -- it's purely a NetBSD-internal software abstraction, like workqueue(9), for software to queue a task and then run it. Normally softint_schedule is called from a hard interrupt handler. Here, you need something in the host to get hard interrupts. For example, on ACPI systems there are ACPI interrupt resources that can be used with acpi_intr_establish; on FDT systems, device nodes have interrupt properties that can be used with fdtbus_intr_establish. How does FreeBSD get x86 mmio intrs on these systems?
Re: VirtIO MMIO for amd64
On 12/16/23 10:25, Emile 'iMil' Heitor wrote: FYI I'm on it, based on Colin Percival's work here https://github.com/freebsd/freebsd-src/blob/main/sys/dev/virtio/mmio/virtio_mmio_cmdline.c I'm getting some results but Firecracker uses MMIO v2 and we only have v1 so there's still quite some work to do. So far I get the block device and the correct geometry is reported [ 1.030] virtio0 at mainbus0 [ 1.030] kernel parameters: console=com root=ld0e virtio_mmio.device=4K@0xd000:5 [ 1.030] viommio: 4K@0xd000:5 [ 1.030] virtio0: block device (id 2, rev. 0x02) [ 1.030] ld0 at virtio0: features: 0x1 [ 1.030] ld0: 30720 KB, 60 cyl, 16 head, 63 sec, 512 bytes/sect x 61441 sectors [ 1.0199904] boot device: ld0 but I get stuck in sys/kern/subr_disk_open.c in opendisk() at: error = VOP_OPEN(tmpvn, FREAD | FSILENT, NOCRED); ideas on what could be locking here? Maybe related: the interrupt handler function I wrote uses softint_establish() as there's no "real" hardware behind this block device, is this the correct way to deal with it? Anybody wanting to have a look https://imil.net/NetBSD/mmio.patch It applies over this branch https://github.com/NetBSDfr/NetBSD-src/tree/GENPVH -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
On 2/23/22 20:26, el16095 wrote: Firecracker informs VMs about MMIO devices by appending to the boot command line a string like this "virtio_mmio.device=4K@0xd000:5" ([virtio_mmio.]device=@:). So, from what I understand I'd need to write glue code that takes this information and uses it to setup the MMIO devices on the VM side the way Firecracker expects it to; and then attach virtio through that, right? FYI I'm on it, based on Colin Percival's work here https://github.com/freebsd/freebsd-src/blob/main/sys/dev/virtio/mmio/virtio_mmio_cmdline.c I'm getting some results but Firecracker uses MMIO v2 and we only have v1 so there's still quite some work to do. -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
On 2/23/22 20:26, el16095 wrote: Firecracker informs VMs about MMIO devices by appending to the boot command line a string like this "virtio_mmio.device=4K@0xd000:5" ([virtio_mmio.]device=@:). So, from what I understand I'd need to write glue code that takes this information and uses it to setup the MMIO devices on the VM side the way Firecracker expects it to; and then attach virtio through that, right? Was there any more work on this topic? -- Emile `iMil' Heitor | https://imil.net
Re: VirtIO MMIO for amd64
Στις 23/02/2022 16:56, Jonathan A. Kollasch έγραψε: On Wed, Feb 23, 2022 at 12:28:36PM +0200, el16095 wrote: I've been working on a NetBSD-based unikernel and trying to make it compatible with the Firecracker VMM. Seeing as Firecracker only supports the MMIO transport method for VirtIO, I wanted to implement MMIO support for my unikernel. I've seen that newer versions of NetBSD than what my source code was using have code for MMIO and specifically that Virtio MMIO support was implemented for ARM in NetBSD 9.0. So, what I want to ask is there similar support for amd64 also and if not are there some basics I could imitate from how it was implemented for ARM to get it working for amd64 also? Alternatively, is there any reading material you can suggest that explains how the device system works in NetBSD, so that I might possibly understand what changes would be needed by myself? There shouldn't be anything preventing our existing Virtio MMIO via ACPI code from working on amd64 in -current. Probably just needs enabling and testing. The "virtio* at acpi?" attachment is not currently included in the i386 or amd64 GENERIC kernels however. I don't even know that Firecracker informs the guest about Virtio MMIO devices via ACPI (or FDT). If not, you'll need to create glue code similar to virtio_acpi.c or virtio_mmio_fdt.c. Thanks, that's very helpful. Firecracker informs VMs about MMIO devices by appending to the boot command line a string like this "virtio_mmio.device=4K@0xd000:5" ([virtio_mmio.]device=@:). So, from what I understand I'd need to write glue code that takes this information and uses it to setup the MMIO devices on the VM side the way Firecracker expects it to; and then attach virtio through that, right? I'm a little uncertain how NetBSD's device discovery works, though, and how I can make it look at the boot commandline for the MMIO devices. Would I need to implement a match/probe function that receives the info from there? And would I then also use the probe function to add the device found to a device list (alldevs, perhaps)? If I make it so, is there something I need to do to let the driver know its match function is actually probing/that it performs indirect configuration? An additional question I have on top of that is, how would I go about mapping a physical address to a virtual one? I believe the address Firecracker provides is a physical one, while what we need for the drivers is a virtual address. I tried to use functions from UVM, but I had some problems with that, so I wondered if that's the standard/only way to do that or if there's something else I could try. -- George Gkanas
Re: VirtIO MMIO for amd64
On Wed, Feb 23, 2022 at 12:28:36PM +0200, el16095 wrote: > I've been working on a NetBSD-based unikernel and trying to make it > compatible with the Firecracker VMM. Seeing as Firecracker only supports the > MMIO transport method for VirtIO, I wanted to implement MMIO support for my > unikernel. > > I've seen that newer versions of NetBSD than what my source code was using > have code for MMIO and specifically that Virtio MMIO support was implemented > for ARM in NetBSD 9.0. So, what I want to ask is there similar support for > amd64 also and if not are there some basics I could imitate from how it was > implemented for ARM to get it working for amd64 also? Alternatively, is > there any reading material you can suggest that explains how the device > system works in NetBSD, so that I might possibly understand what changes > would be needed by myself? There shouldn't be anything preventing our existing Virtio MMIO via ACPI code from working on amd64 in -current. Probably just needs enabling and testing. The "virtio* at acpi?" attachment is not currently included in the i386 or amd64 GENERIC kernels however. I don't even know that Firecracker informs the guest about Virtio MMIO devices via ACPI (or FDT). If not, you'll need to create glue code similar to virtio_acpi.c or virtio_mmio_fdt.c.
VirtIO MMIO for amd64
I've been working on a NetBSD-based unikernel and trying to make it compatible with the Firecracker VMM. Seeing as Firecracker only supports the MMIO transport method for VirtIO, I wanted to implement MMIO support for my unikernel. I've seen that newer versions of NetBSD than what my source code was using have code for MMIO and specifically that Virtio MMIO support was implemented for ARM in NetBSD 9.0. So, what I want to ask is there similar support for amd64 also and if not are there some basics I could imitate from how it was implemented for ARM to get it working for amd64 also? Alternatively, is there any reading material you can suggest that explains how the device system works in NetBSD, so that I might possibly understand what changes would be needed by myself? -- George Gkanas
