Hi everyone,
Over the last couple of months I've been working on implementing PCI device
hotplugging in bhyve and would like to get some feedback on the changes.
The patch series from D54641 to D54656 implements an ACPI-based
hotplugging mechanism
that is accessible through an nvlist(9)-based IPC protocol.
Users can add to or remove an emulated PCI device from a running virtual
machine
by sending an appropriately crafted nvlist(9) object to a UNIX socket
located at '/var/run/bhyve/<VM_name>'.
The patch series also adds new command-line options to 'bhyvectl'
('--pci-{add,remove}') to facilitate these operations.
Since the implementation ended up touching a lot of files and adding a
lot of
changes to the PCI emulation layer, I'll try to give a brief overview of the
problems each change is meant to solve and outline the major changes in
this patch series.
Aside from implementing the ACPI-based hotplugging mechanism, adding PCI
hotplugging to bhyve required solving the following problems:
1) bhyve's PCI BAR allocator can't allocate and free BARs during runtime,
2) bhyve is sandboxed, which makes it impossible to initialize certain
emulated
PCI devices during runtime since they might need to open files
during initialization,
3) Allowing users to request hotplug or hotremove operations from a
running
bhyve instance.
The first problem was solved using 'libuvmem(3)', a userspace port of the
'vmem(9)' general-purpose allocator. This allows us to manage resources
owned by
each virtual PCI bus during runtime. The second problem was solved by
delegating
parts of the PCI device initialization to 'bhyvectl'. Under this scheme,
'bhyvectl'
opens all required files and sends the appropriate file descriptors to
bhyve over
a UNIX socket. To avoid unnecessary code duplication, parts of the bhyve's
device-specific initialization routines were moved to 'libbhyve', a new
private library.
The third problem was solved by generalizing bhyve's VM snapshotting IPC
thread.
This allows end users to invoke bhyve's PCI hotplugging routines through an
nvlist(9)-based IPC protocol.
In summary, here's a brief overview of the changes made by this patch
series.
- Replacing bhyve's static PCI BAR allocation algorithm with
'libuvmem(3)' (D54641),
- Adding ACPI methods that allow us to inject hotplugging events into
the guest (D54642 - D54646),
- Generalizing bhyve's IPC thread and introducing an nvlist(9)-based IPC
protocol (D54650),
- Refactoring and moving parts of the device initialization and
configuration code to 'libbhyve' (D54648),
- Adding hotplugging IPC commands to 'bhyve' and the corresponding
'--pci-{add,remove}' options to 'bhyvectl' (D54651 - D54653),
- Making the 'e1000' and 'virtio-net' network devices hotpluggable -
(D54654 - D54656).
The network devices were chosen as a proof of concept. Other devices will
be added after this patch series gets reviewed.
Please feel free to leave comments on any of the reviews from D54641 to
D54656.
Thanks,
Bojan
