> On Feb 25, 2019, at 7:29 PM, Mike Larkin <[email protected]> wrote:
>
> On Sun, Feb 24, 2019 at 12:21:24PM -0500, Dave Voutila wrote:
>> I've been experimenting with implementing something like vmmci(4) for
>> Linux guests. It's started to prove useful to myself so maybe others
>> will benefit, even though there are currently some caveats[1].
>>
>> https://github.com/voutilad/virtio_vmmci
>>
>> My primary use case is keeping some Linux guests constantly running in
>> the background under the vmm(4) hypervisor on my laptop that sees quite
>> a few suspend (zzz)/resume cycles throughout my day.
>>
>> It currently consists of 2 Linux kernel modules, one representing a
>> virtio_pci implementation handling the quirks[2] of how vmd(8) exposes
>> the vmm control device and the other module is the actual virtio driver
>> implementation.
>>
>> I've tested it with a stock Ubuntu 18.0.4 install as well as with a
>> newer v4.20.12 tweaked Linux kernel[3] and so far seems to build and
>> work just fine using an OpenBSD snapshot from Monday 18 Feb as the host.
>>
>> It doesn't yet catch shutdown or restart events, but that's my next goal
>> since I'd like to make sure I get clean shutdowns via `vmctl stop`.
>>
>> -Dave
>>
>> [1]: https://github.com/voutilad/virtio_vmmci#current-state--known-caveats
>> [2]: https://github.com/voutilad/virtio_vmmci#learnings-from-virtio-hacking
>> [3]: https://github.com/voutilad/linux
>>
>
> Looks like a good start!
>
> For what it's worth, we should probably allocate a real virtio device number
> from Redhat or whoever controls that. I have an old email in my inbox with a
> few contact names, but I never got around to following up. That would fix the
> problem of the "stolen" virtio ID.
>
> We chose to implement what looks like a PCI hostbridge but used a fake PCI
> VID/PID for this. The reason for doing it this way is that if you say you are
> a host bridge that is well-known, you have to implement all the bug-for-bug
> compatibility that comes with that.
>
> I'll take a look at the register read/write thing, I'm not sure I entirely
> understand the issue you saw.
>
> -ml
I'll try to avoid too much Linux detail since this is misc@openbsd,
but it seems to be a difference related to the underlying Linux virtio
pci functions used for reading based on if your device is picked up as
a "modern" or "legacy" device.:
- virtio_pci_modern[1]: uses a similar approach to how
OpenBSD's virtio pci implementation works for reading config
registers where there are different routines for reading different
amounts of data.
- virtio_pci_legacy[2]: reads a byte using Linux's ioread8 from
the given register address, then reads a byte from address+1,
etc. until it reads up to the amount of data requested.
Given how vmd/virtio.c:vmmci_io [3] is implemented it expects a
read/write to specific register addresses, with logical gaps between
them based on the size of the data returned:
...
int
vmmci_io(int dir, uint16_t reg, uint32_t *data, uint8_t *intr,
void *unused, uint8_t sz)
{
*intr = 0xFF;
if (dir == 0) {
switch (reg) {
case VIRTIO_CONFIG_DEVICE_FEATURES:
case VIRTIO_CONFIG_QUEUE_SIZE:
case VIRTIO_CONFIG_ISR_STATUS:
log_warnx("%s: illegal write %x to %s",
__progname, *data, virtio_reg_name(reg));
break;
...
I had to change my driver to use the "modern" behavior even though
all OpenBSD virtio devices appear as "legacy" to the Linux virtio pci
framework. (So far writes using the legacy method are fine.)
I'm not convinced this warrants changes to vmd/virtio.c since this is
like fringe use case after fringe use case and all the other virtio
devices vmd(8) implements work. The VMM Control device is a bit of the
oddball because it's not using virtio queues to communicate and
instead relies on interrupts to notify the driver to go check the
config registers.
Which brings me to my last part: getting interrupts to work on my
driver. I figured it out this morning after ripping out MSI/MSI-X and
virtio queue stuff and have something working now and my driver picks
up the events listened for by vmmci(4). Shouldn't be too far now from
having a nice, clean shutdown or restart via `vmctl stop ubuntu`. Will
share when it’s ready if systemd doesn’t prematurely end my sanity
first.
-dv
[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/virtio/virtio_pci_modern.c#n193
[2]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/virtio/virtio_pci_legacy.c#n50
[3]: https://github.com/openbsd/src/blob/master/usr.sbin/vmd/virtio.c#L1667
