Hi Nikhil, On Tuesday, 22.05.2018 at 13:18, nikhil ap wrote: > Currently solo5/ukvm supports only one NIC. I wanted to know your thoughts > on supporting multiple NICs. This is really important and essential > requirement when trying to run on IncludeOS unikernels which is heavily > focused on networking and NFV. > > After talking to Ricardo, I understand that the difficulty is in coming up > with a new API. I wanted to start off the discussion.
The issue is not so much working out what the guest-side API (i.e. that in solo5.h) should be, but keeping the conceptual integrity of how Solo5 (kernel/bindings), ukvm (monitor/tender) and application (unikernel) fit together. This will require a digression into the project's history, so please bear with me. One of the original goals for ukvm, outlined in Dan and Ricardo's original 2016 paper on the subject [1] was to "specialize" the monitor to match the running unikernel. Among other things, the benefits of this approach are: 1.1. We ensure that the trusted tender code is minimal and does not contain or enable any "unused" interfaces. Such interfaces are a well-known source of security issues (see e.g. the now-famous QEMU floppy-drive bug). 1.2. At the same time, we ensure that operation of the tender is _coupled_ to the unikernel it is running. In other words, in order for an operator/orchestrator to run a ukvm-based unikernel today, *all* the host resources (devices) the unikernel requires to operate are configured at start-up. You cannot [2] run a ukvm-bin configured with a "UKVM_MODULE_NET" without also specifying with --net which host resource the guest device should be bound to. Both of these are intentional and constitute a large part of what makes Solo5/ukvm unique compared to many similar systems that exist today. The way they are accomplished today is by _coupling_ the tender to the unikernel at _build time_. We (Dan, Ricardo and I) have known for quite some time that the build-time coupling of the tender and unikernel -- while it seemed like a good idea at the time -- is not practical for use in real-world scenarios, for several reasons: 2.1. "Users" (producers of "untrusted" applications that run under a Solo5 tender) should not be providing their own "trusted" tender binary. This goes against any security model that e.g. a public/private cloud deploying such applications would want to use in practice [2]. 2.2. Build-time coupling is totally inflexible for multi-platform host OS support, as it requires the "user" to build all possible combinations of tender binaries for all targets they might want to deploy their application on. For example, a "user" building their unikernels on Linux but deploying on FreeBSD has no way [3] to produce a "correct" tender binary. 2.3. Build-time coupling cannot cleanly support N instances of a resource (be that NICs, or block devices) while at the same time staying consistent with point (1.2) above. The question is, how can we resolve the above points going forward, keeping in mind (1.1) and (1.2)? My preferred approach (discussed with Dan & Ricardo several times, but not in public) would be to replace the current model of build-time coupling with run-time coupling. The main points of a "back of the napkin" implementation would be something like this: 3.1. The application binary, supplied by the "user", would contain a "manifest" (e.g. inside a special ELF note) declaring which resources (NICs, block storage, future IPC, etc.) it requires to operate. 3.2. The "manifest" would include information regarding the exact ABI version that the application requires, so we could actually provide a stable ABI contract between the tender and application, which we do not do today. 3.3. It follows that the tender can now be supplied separately by the "operator". At start-up, it would load the manifest and use it to determine which modules to enable [4], based on this list it would also require the "operator" to configure all host resources for all enabled modules. Getting to this point requires a significant amount of development work, and time, however I think its absolutely necessary for Solo5 to be widely usable in practice. If you want to get support for multiple NICs before then, I'm interested in interim compromise proposals (prose first, not code!) that consider the changes needed to all components involved (tender, bindings, application) and can work with the current source-time coupling while not being too gross of a hack. I've thought of several options, but couldn't find one that passed. Rather than presenting them right now, I'd prefer to hear ideas from others on what approaches might work. Thanks for reading, and I hope this explains why it's not "just" an issue of adding an index to the guest-side APIs. Please ask questions. -mato [1] https://www.usenix.org/system/files/conference/hotcloud16/hotcloud16_williams.pdf [2] There are people playing various tricks today to make this work, it's not hard. You know who you are. [3] Without dealing with cross-toolchains. This gets even worse if dealing with different executable file formats, which is one of the reasons the initial macos Hypervisor.framework port of ukvm was never merged. [4] "Enable" might mean "load" rather than just "set a variable". It should be possible to implement a tender that does not even contain the code for modules that are not enabled.
