Hi Martin,
On Tue, Apr 24, 2018 at 1:44 PM, Martin Lucina <mar...@lucina.net> wrote: > Hi Nikhil, > > On Friday, 20.04.2018 at 20:26, nikhil ap wrote: > > Hi Guys, > > > > Summarising the discussions we've had so far and the work I've been > doing: > > Thank you for this summary and for doing these experiments. The numbers are > very impressive. > > For those who were not on the private email thread about this, Nikhil has > been working on a minimal PoC implementation of the Solo5/ukvm network APIs > with a shared memory transport based on the "shmstream" protocol used by > the Muen Separation Kernel. > > > - Implemented network interface using shmstream (shared memory) in > order > > to reduce the number of hypercalls and thus reduce the number of > VMEntries > > and VMExits. > > - Separate IO thread is implemented in ukvm-bin to read/write packets > > from the shared memory. > > - IO thread currently supports polling mode and event mode. > > What exactly do you mean by "polling mode"? Just using poll() on > non-blocking sockets similarly to what the current ukvm implementation of > solo5_yield() does? > Polling mode: IO thread keeps polling for packets on tapfd and shmstream and if there is nothing, sleeps for a millisecond. Hence the CPU consumption is high even when there is no data. Event mode: IO thread waits for events using epoll. Only wakes up when there is actually data to be read from tapfd and shmstream. When there is no data CPU usage is 0% but it has more VMEXITs than polling mode because the solo5 kernel need to notify when the packets are queued on shmstream using the hypercall. Hence queuing multiple packets onto shmstream and notifying it with solo5_net_flush() will reduce the number of hypercalls. > > > - Event-driven model is implemented using eventfds and io-thread waits > > for events using epoll. > > This would require a portability layer to run on the BSDs. If I'm reading > the figures you sent correctly, it looks like polling performs better > anyway? > Polling mode is just to figure out what is the best we can do but I'm not sure if it is suitable for production. The goal is to achieve a very good performance which is comparable to qemu and which is not CPU intensive. > > > - The applications could run both the modes without any changes to > their > > APIs. > > - Currently shmstream mode can be set with --shm option. > > - Ex: ./ukvm-bin --net=tap100 --shm=poll test_ping_serve.ukvm > > In the grand scheme of things, I'd like to eventually replace the current > implementation with a shmstream-based one entirely, if it proves to be > portable (host OS and arch-wise) and secure enough. So, it would not be > user-selectable. However, there are also other things that need to happen > before that (see below). > > > - However, in case of event-mode, for better performance, the > > application can chose to notify ukvm-bin after queuing *all* the > packets in > > the shared memory instead of a single packet transmit by using a new > solo5 > > public APIs: solo5_net_queue() and solo5_net_flush(); > > The same thing could presumably be accomplished with a writev()-style API, > or is this doing something different? With solo5_net_queue() API, we can queue packets in shmstream without notifying ukvm-bin to read from it and signal ukvm-bin with solo5_net_flush(). > What is the performance gain in > practice? In the figures you sent, is that with or without this > optimization? It is with optimisation. Although it doesn't matter since I have tested only receiving packets. I'll test transmit side also and then update. > > > - Solo5 performance was tested with IncludeOS (IncludeOS had to be > > modified to address the recent API changes in solo5) and with UDP > traffic. > > - Summarising the results below > > Do you have any statistics comparing (host) CPU load for the various tests? > CPU load is mentioned in my initial email which is on the host or do you require something else? > > Next steps: > > I'd like to push out a formal release of Solo5 that integrates the work > done over the past year which does not have a proper formal release > (FreeBSD vmm support, ARM64 support) and some of the work which is "ready > to merge" ASAP (OpenBSD vmm support, possibly your work on guest-side core > dumps). This would also include the renaming of the various Solo5 > components as discussed in #172. > > Once that is done we will then have time to look at integrating your > shmstream-based network implementation, and, as I wrote above, ideally to > replace the current implementation entirely. > > In keeping with the minimalist design of Solo5 and ukvm, rather than > providing several different network implementations, or variants of one > (polling vs. event-driven, etc.) I'd prefer that we work towards choosing > one single variant to use and support [as a default], otherwise we run the > risk of going down the "QEMU-style" route of a proliferation of different > optional components, which is both hard to understand for users and hard to > comprehensively test and secure. > > So, I suggest that you continue to iterate on this work, ideally in public, > and submit a PR after the next Solo5 release. > Any ETA on the next solo5 release? > > Cheers, and thanks again, > > -mato > -- Regards, Nikhil
