Hmm... I wonder if paravirtualization would work better with something
like my blk2scsa framework (see PSARC 2007/654). Alternatively, maybe
a separate block driver rather than modifying cmdk?
The idea that we have to touch each driver that needs to support
paravirtualization seems kind of clumsy to me. A lot of the complexity
in extant drivers is to deal with various hardware bugs and workarounds,
and I fear that this approach just further complicates already complex
drivers, and maybe also robs PV drivers of some their performance as well?
But it sounds, from the materials below, like maybe for cmdk's specific
case this isn't too bad, and there maybe upgrade issues that are avoided
by this approach?
-- Garrett
Tim Marsland wrote:
> Template Version: @(#)sac_nextcase 1.64 07/13/07 SMI
>
> 1. Introduction
> 1.1. Project/Component Working Name:
> Paravirtualized Drivers for Fully Virtualized xVM Domains
>
>
> 1.2. Name of Document Author/Supplier:
> Author: Nils Nieuwejaar
>
> 1.3 Date of This Document:
> 04 November, 2007
>
> 4. Technical Description
>
> 4.1 Summary
>
> The Xen hypervisor and the port of Solaris to that hypervisor are
> described in PSARC/2006/260. That case is considered required
> pre-reading for this.
>
> The Solaris port to the Xen hypervisor included disk and network
> drivers for paravirtualized (PV) domUs. These PV drivers provide
> standard disk and network interfaces to Solaris, but communicate
> with back-end drivers in the control domain (dom0) rather than
> interacting directly with hardware.
>
> When running in a fully virtualized (HVM) domain, Solaris continues
> to use standard device drivers. These drivers believe they are
> interacting with hardware, but are actually interacting with
> emulated devices that run in dom0. These emulated physical devices
> deliver significantly lower performance than the corresponding
> virtual devices in a PV domain. For example, the emulated network
> device may run at 1/10th the speed of a virtual device.
>
> This project proposes to provide the infrastructure necessary to
> allow the existing Solaris PV drivers to function in HVM domains,
> delivering significantly higher performance to applications running
> in those domains.
>
> We intend to backport these PV drivers to Solaris 10, enabling it
> to be a performant guest operating system. Therefore, we are
> requesting micro/patch binding for this project.
>
> 4.2 Proposal
>
> The project is comprised of several components. There is one new
> driver that provides the infrastructure necessary to allow software
> in an HVM domain to communicate with the hypervisor and the control
> domain. There are also minor modifications to the existing
> drivers, that will allow them to plug into the new infrastructure.
>
> 4.2.1 xpv
>
> xpv is a new leaf driver that binds to Xen's PCI ID
> (pci5853,1). This driver provides all the plumbing needed for
> PV drivers to drill through the bottom of a fully virtualized
> domain, and interact directly with the hypervisor.
>
> The driver uses HVM-specific hypercalls to:
>
> o map in the 'Xen shared info' page, which is used for
> communication between the hypervisor and OS.
>
> o map pages for grant tables, which are used to enable
> temporary transfers of page ownership between domains
>
> o register a single interrupt with the hypervisor, which it
> will trigger when there are events pending for the OS to
> process.
>
> After initialization, the driver's primary responsibility is to
> receive emulated 'physical' interrupts from the hypervisor when
> events are pending, and then forward those event notifications
> to the correct PV driver(s).
>
> 4.2.2 Modified Drivers
>
> cmdk is an existing Solaris driver that provides a common
> framework for block devices. This driver presents a simple
> high-level disk model to Solaris while abstracting away the
> distinctions between disks, CD drives, etc.
>
> This project modifies cmdk to be PV-aware. When loaded in an
> HVM domain, this driver uses the Layered Driver framework to
> forward all disk requests to the PV xdf driver rather than to
> the standard driver(s) for emulated physical devices. This
> forwarding mechanism allows us to add the PV drivers after a
> domain has already been installed, without having to modify the
> path to the boot device.
>
> In addition, the paravirtualized xdf, xnf, xpvd, and xenbus
> drivers must all be modified to register event handlers with a
> new 'xpv' driver, rather than registering interrupt handlers
> with Solaris. Other than this event delivery mechanism, all of
> a driver's interaction with the hypervisor and control plane is
> exactly the same in both PV and HVM domains.
>
> The scope of these changes is on the order of about a dozen
> lines per driver.
>
> 4.3 User Experience
>
> 4.3.1 Background
>
> Xen has adopted two very different models for supporting PV
> devices in HVM domains. These models are built into the
> architecture of Xen itself, and are not specific to Sun's
> implementation.
>
> For block devices, each device configured in a domU has two
> interfaces: an emulated ATA device on the PCI bus, and an
> explicit PV interface accessible through the Xenbus. The two
> interfaces provide access to the same storage on the back end.
>
> For network devices, each NIC is configured in dom0 to be
> either PV or HVM. If a device is configured to be HVM, Xen
> provides an emulated RealTek NIC on the PCI bus. If it is
> configured to be PV, then it provides an explicit PV interface
> on the Xenbus. A single NIC cannot be configured in both modes
> at once.
>
> 4.3.2 Initial Installation
>
> When the PV drivers are integrated into the WOS, they will
> function as any other block or network driver.
>
> When a domU is installed, the disks will automatically and
> transparently be accessed using the PV interfaces. Users will
> install to /dev/dsk/cXdYsZ as they do now. All the normal
> tools, such as format, prtvtoc, and fdisk will work as
> expected.
>
> When the installer probes for network devices, any PV NICs
> configured into the domain will be made available as xnf0
> through xnf<N>, just as (for example) Intel NICS are made
> available as e1000g0 through e1000g<N>.
>
> 4.3.3 Upgrades or post-install additions
>
> When the PV drivers are added to an existing domain, our
> modifications to the cmdk driver will allow the PV disk drivers
> to fall into place transparently. When the driver is first
> installed, the system will continue to access disks using the
> existing emulated metal interfaces. When the domain is
> rebooted, the disks will automatically be accessed through the
> PV interfaces. The domain's bootpath remains unchanged, so the
> transition from HVM to PV should be completely transparent to
> the user.
>
> For network devices, however, the user experience will be less
> satisfying. As a NIC must be explicitly configured in dom0 to
> be either PV or HVM, simply installing the PV drivers will not
> deliver PV performance to the user. The domU must be reconfig-
> ured to contain PV NICs rather than HVM. When the domain is
> reconfigured, the 'brand' of the NIC changes from rtls (i.e.
> "RealTek") to xnf. Since Solaris binds the identity of a
> system to its NIC name, the user will have to sys-unconfig and
> reconfigure the domain.
>
> A natural question is: can't we transparently remap the rtls
> devices to xnf devices under the hood (as we do for disks),
> making this reconfiguration step go away. Unfortunately, no.
>
> Whether a NIC is configured to be PV or HVM is not simply a
> cosmetic issue. When Xen instantiates an HVM NIC for a domain,
> it does not create a back-end device that is capable of
> interacting with a front-end PV driver. Even if the domU had
> sufficient information about the back-end configuration to
> understand how this remapping should be done, there is simply
> nothing in the dom0 for our PV drivers to talk to.
>
> While this upgrade experience is certainly less smooth than one
> would hope, it is not dissimilar to the experience of
> upgrading physical hardware. Replacing a disk in a physical
> machine is a simple process, and the device naming is not
> affected by the brand of the hard drive. Replacing an Intel
> NIC with a Broadcom NIC, on other hand, requires exactly the
> same sys-unconfig as that required by replacing a fully
> virtualized NIC with a paravirtualized NIC.
>
> 4.4 Interface Table
>
> Imported Interfaces
> ---------------------------------------------------------------------
> | Interface Level Comments
> |--------------------------------------------------------------------
> | Layered Driver Evolving
> | Framework
> |
> | Xen Interfaces External
> | Hypercall interface
> | Shared page format
> | Xenbus protocol
> |
> | PCI ID pci5853,1 External Xen's registered
> | PCI ID
> |--------------------------------------------------------------------
>
> Exported Interfaces
> ---------------------------------------------------------------------
> | Interface Level Comments
> |--------------------------------------------------------------------
> | ec_bind_evtchn_to_handler() Project Used by the PV
> | ec_unbind_evtchn() private drivers to plug into
> | hypervisor_mask_event() the new PV-on-HVM
> | hypervisor_unmask_event() framework.
> |--------------------------------------------------------------------
>
> 4.4 References
>
> [1] PSARC 2006/260: Solaris on Xen
>
> [2] PSARC 2001/769: LDF: Layered Driver Framework
>
> 6. Resources and Schedule
> 6.4. Steering Committee requested information
> 6.4.1. Consolidation C-team Name:
> ON
> 6.5. ARC review type: FastTrack
>
> 6. Resources and Schedule
> 6.4. Steering Committee requested information
> 6.4.1. Consolidation C-team Name:
> ON
> 6.5. ARC review type: FastTrack
> 6.6. ARC Exposure: open
>
>
