Hi Guys,
Summarising the discussions we've had so far and the work I've been doing:
- Implemented network interface using shmstream (shared memory) in order
to reduce the number of hypercalls and thus reduce the number of VMEntries
and VMExits.
- Separate IO thread is implemented in ukvm-bin to read/write packets
from the shared memory.
- IO thread currently supports polling mode and event mode.
- Event-driven model is implemented using eventfds and io-thread waits
for events using epoll.
- The applications could run both the modes without any changes to their
APIs.
- Currently shmstream mode can be set with --shm option.
- Ex: ./ukvm-bin --net=tap100 --shm=poll test_ping_serve.ukvm
- However, in case of event-mode, for better performance, the
application can chose to notify ukvm-bin after queuing *all* the packets in
the shared memory instead of a single packet transmit by using a new solo5
public APIs: solo5_net_queue() and solo5_net_flush();
- Solo5 performance was tested with IncludeOS (IncludeOS had to be
modified to address the recent API changes in solo5) and with UDP traffic.
- Summarising the results below
+--------------------------------------------------+----------------------------------------------------------------------+
| Experiment |
PPS with 1300 byte packets as receiver |
+--------------------------------------------------+----------------------------------------------------------------------+
| With Qemu and vhost=on | 460k
|
+--------------------------------------------------+----------------------------------------------------------------------+
| With Qemu and vhost=off | 310k
|
+--------------------------------------------------+-------------------------------------------------
---------------------+
| With solo5/ukvm |
160k |
+--------------------------------------------------+-------------------------------------------------
---------------------+
| With solo5/ukvm withseccomp | 440k
|
+--------------------------------------------------+-------------------------------------------------
---------------------+
| With solo5/ukvm withshmstream and eventthread | 435k
|
+--------------------------------------------------+-------------------------------------------------
---------------------+
| With solo5/ukvm withshmstream and polling thread | 490k
|
+--------------------------------------------------+-------------------------------------------------
---------------------+
UDP Performance stats collected after running each test for 30 seconds
+---------------------------------------+-------------------+--------------------+
---------------------------+
| Model |
VMentries/VMexits | CPU usage |
+---------------------------------------+-------------------+--------------------+
---------------------------+
| Solo5/ukvm today | 8.7 Million
| 100% |
+---------------------------------------+-------------------+--------------------+
---------------------------+
| Solo5 with shmstream and polling | 215K |
200% |
+---------------------------------------+-------------------+--------------------+
---------------------------+
| Solo5 with shmstream and event-driven | 260K | 160%
|
+---------------------------------------+-------------------+--------------------+
---------------------------+
| Qemu with Vhost | 20 Million
| 100% + 50% (Vhost) |
+---------------------------------------+-------------------+--------------------+
---------------------------+
TODO: Support IO thread with hybrid mode. I.E with both event driven and
polling mode. Similar to NAPI.
--
Regards,
Nikhil
