On Wed, Apr 11, 2018 at 3:29 AM, Waldek Kozaczuk <jwkozac...@gmail.com> wrote:
> Last week I have been trying to hack OSv to run on hyperkit and finally I > managed to execute native hello world example with ROFS. > > Here is a timing on hyperkit/OSX (the bootchart does not work on hyperkit > due to not granular enough timer): > > OSv v0.24-516-gc872202 > Hello from C code > > *real 0m0.075s * > *user 0m0.012s * > *sys 0m0.058s* > > command to boot it (please note that I hacked the lzloader ELF to support > multiboot): > > hyperkit -A -m 512M \ > -s 0:0,hostbridge \ > -s 31,lpc \ > -l com1,stdio \ > -s 4,virtio-blk,test.img \ > -f multiboot,lzloader.elf > Impressive! How hard is it to setup hyperkit on osx, just brew install? > > Here is a timing on QEMU/KVM on Linux (same hardware - my laptop is setup > to triple-boot Ubuntu 16/Mac OSX and Windows): > > OSv v0.24-510-g451dc6d > 4 CPUs detected > Firmware vendor: SeaBIOS > bsd: initializing - done > VFS: mounting ramfs at / > VFS: mounting devfs at /dev > net: initializing - done > vga: Add VGA device instance > virtio-blk: Add blk device instances 0 as vblk0, devsize=8520192 > random: intel drng, rdrand registered as a source. > random: <Software, Yarrow> initialized > VFS: unmounting /dev > VFS: mounting rofs at /rofs > VFS: mounting devfs at /dev > VFS: mounting procfs at /proc > VFS: mounting ramfs at /tmp > disk read (real mode): 28.31ms, (+28.31ms) > uncompress lzloader.elf: 49.63ms, (+21.32ms) > TLS initialization: 50.23ms, (+0.59ms) > .init functions: 52.22ms, (+1.99ms) > SMP launched: 53.01ms, (+0.79ms) > VFS initialized: 55.25ms, (+2.24ms) > Network initialized: 55.54ms, (+0.29ms) > pvpanic done: 55.66ms, (+0.12ms) > pci enumerated: 60.40ms, (+4.74ms) > drivers probe: 60.40ms, (+0.00ms) > drivers loaded: 126.37ms, (+65.97ms) > ROFS mounted: 128.65ms, (+2.28ms) > Total time: 128.65ms, (+0.00ms) > Hello from C code > VFS: unmounting /dev > VFS: unmounting /proc > VFS: unmounting / > ROFS: spent 1.00 ms reading from disk > ROFS: read 21 512-byte blocks from disk > ROFS: allocated 18 512-byte blocks of cache memory > ROFS: hit ratio is 89.47% > Powering off. > > *real 0m1.049s* > *user 0m0.173s* > *sys 0m0.253s* > > booted like so: > > qemu-system-x86_64 -m 2G -smp 4 \ > > -device virtio-blk-pci,id=blk0,bootindex=0,drive=hd0,scsi=off \ > > -drive > file=/home/wkozaczuk/projects/osv/build/last/usr.img,if=none,id=hd0,cache=none,aio=native > \ > > -enable-kvm -cpu host,+x2apic \ > > -chardev stdio,mux=on,id=stdio,signal=off \ > > -mon chardev=stdio,mode=readline > > -device isa-serial,chardev=stdio > > > In both cases I am not using networking - only block device. BTW I have > not tested how networking nor SMP on hyperkit with OSv. > > So as you can see* OSv is 10 (ten) times faster* on the same hardware. I > am not sure if my results are representative. But if they are it would mean > that QEMU is probably the culprit. Please see my questions/consideration > toward the end of the email. > > Anyway let me give you some background. What is hyperkit? Hyperkit ( > https://github.com/moby/hyperkit) is a fork by Docker of xhyve ( > https://github.com/mist64/xhyve) which itself is a port of bhyve ( > https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html) - > hypervisor on FreeBSD. Bhyve architecture is similar to that of KVM/QEMU > but QEMU-equivalent of bhyve is much lighter and simpler: > > "The bhyve BSD-licensed hypervisor became part of the base system with > FreeBSD 10.0-RELEASE. This hypervisor supports a number of guests, > including FreeBSD, OpenBSD, and many Linux® distributions. By default, > bhyve provides access to serial console and does not emulate a graphical > console. Virtualization offload features of newer CPUs are used to avoid > the legacy methods of translating instructions and manually managing memory > mappings. > > The bhyve design requires a processor that supports Intel® Extended Page > Tables (EPT) or AMD® Rapid Virtualization Indexing (RVI) or Nested Page > Tables (NPT). Hosting Linux® guests or FreeBSD guests with more than one > vCPU requires VMX unrestricted mode support (UG). Most newer processors, > specifically the Intel® Core™ i3/i5/i7 and Intel® Xeon™ E3/E5/E7, support > these features. UG support was introduced with Intel's Westmere > micro-architecture. For a complete list of Intel® processors that support > EPT, refer to http://ark.intel.com/search/advanced?s=t& > ExtendedPageTables=true. RVI is found on the third generation and later > of the AMD Opteron™ (Barcelona) processors" > > Hyperkit/Xhyve is a port of bhyve but targets Apple OSX as a host system > and instead of FreeBSD vmm kernel module uses Apple hypervisor framework ( > https://developer.apple.com/documentation/hypervisor). Docker, I think, > forked xhyve to create hyperkit in order to provide lighter alternative of > running Docker containers on Linux on Mac. So in essence hyperkit is a > component of Docker for Mac vs Docker Machine/Toolbox (based on > VirtualBox). Please see for details there - https://docs.docker.com/ > docker-for-mac/docker-toolbox/. > > How does it apply to OSv? It only applies if you want to run OSv on Mac. > Now the only choice is QEMU (dog slow because no KVM) or VirtualBox (pretty > fast once OSv is up but it takes long time to boot and has other > configuration quirks). Based on my experiments hyperkit becomes very > compelling new alternative. > > Reletely you maybe aware of uKVM (https://github.com/Solo5/solo5) - very > light hypervisor to run clean-state unikernels like MirageOS or IncludeOS. > There is an OSv issue - https://github.com/cloudius- > systems/osv/issues/886- and corresponding one on uKVM - > https://github.com/Solo5/solo5/issues/249 - to track what it would take > to boot OSv on it. It turns our (read issues for details) that in current > form uKVM is too minimalistic for OSv. For example there is no interrupts. > The experiment with hyperkit made me think that it would be nice if there > an alternative to QEMU on Linux - simpler and lighter than qemu but not as > minimalistic as uKVM - something equivalent to bhyve for Linux. > > Waldek > > PS. I have created an issue https://github.com/ > cloudius-systems/osv/issues/948 to track what it would take to make OSv > run on hyperkit. > > > -- > You received this message because you are subscribed to the Google Groups > "OSv Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to osv-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Asias -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.