One more thing - our serial console is dreadfully slow (https://github.com/cloudius-systems/osv/issues/921). So I think even printing bootchart info slows things by 3-5 ms. Without bootchart the 17 ms number drops to 12-11 ms.
On Saturday, January 5, 2019 at 10:32:18 PM UTC-5, Waldek Kozaczuk wrote: > > > > On Saturday, January 5, 2019 at 4:09:23 AM UTC-5, דור לאור wrote: >> >> Great stuff Waldek! Is unix socket the normal way to pass parameters for >> Firecracker? >> > I think so. There is also experimental vsock interface but I am not > familiar with it and not sure what purpose of it would be ( > https://github.com/firecracker-microvm/firecracker/issues/650). > >> >> How's the boot speed vs Qemu? >> > I have posted some numbers in my other email to the group but here are > some more details. > > First of all my numbers come from running tests on my 5-years old MacBook > pro that I have been using for all my OSv development in last three years. > It is 4-core 2.3 GHz i7 machine. Not sure how it compares to newer models > but given Moore's law does not work any more it might be still pretty fast. > > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 8 > On-line CPU(s) list: 0-7 > Thread(s) per core: 2 > Core(s) per socket: 4 > Socket(s): 1 > NUMA node(s): 1 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 70 > Model name: Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz > Stepping: 1 > CPU MHz: 898.037 > CPU max MHz: 3500.0000 > CPU min MHz: 800.0000 > BogoMIPS: 4589.68 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 6144K > L4 cache: 131072K > > Additionally firecracker startup/configuration is more fine-grained than > QEMU which does everything in one shot (start VMM process, configure > resources, start guest, etc). With firecracker it is broken down like > follows: > > 1. Start VMM process which listen on socket for API calls (I have not > measured it but seems very fast). > 2. Make API call to: > - create instance that specified number of vCPUs, memory and kernel > loader file path > - configure block device > - configure networking device > - all these calls seem to take less than 1ms and can be executed in > any order it seems > 3. Make API call to start instance that eventually starts guest > > So here are some log snippets (2,3) and OSv bootchart from running > native-example rofs image (no networking): > > Add block device: > 2019-01-05T20:21:44.*096*617431 [anonymous-instance:INFO:api_server/src/ > http_service.rs:599] The API server received a synchronous Put request on > "/drives/rootfs" with body "{n "drive_id": "rootfs",n "path_on_host": > "/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device": > false,n "is_read_only": falsen }". > 2019-01-05T20:21:44.*096*659174 [anonymous-instance:INFO:api_server/src/ > http_service.rs:565] The synchronous Put request on "/drives/rootfs" with > body "{n "drive_id": "rootfs",n "path_on_host": > "/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device": > false,n "is_read_only": falsen }" was executed successfully. Status code: > 204 No Content. > > Create instance: > 2019-01-05T20:21:44.*1085*43339 [anonymous-instance:INFO:api_server/src/ > http_service.rs:599] The API server received a synchronous Put request on > "/boot-source" with body "{n "kernel_image_path": > "/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n > "boot_args": "--bootchart /hello"n }". > 2019-01-05T20:21:44.*108*584295 [anonymous-instance:INFO:api_server/src/ > http_service.rs:565] The synchronous Put request on "/boot-source" with > body "{n "kernel_image_path": > "/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n > "boot_args": "--bootchart /hello"n }" was executed successfully. Status > code: 204 No Content. > > Start instance that starts guest and terminates the process eventually: > 2019-01-05T20:21:44.119820357 [anonymous-instance:INFO:api_server/src/ > http_service.rs:599] The API server received a synchronous Put request on > "/actions" with body "{n "action_type": "InstanceStart"n }". > 2019-01-05T20:21:44.119837722 [anonymous-instance:INFO:vmm/src/lib.rs:1104] > VMM received instance start command > 2019-01-05T20:21:44.119903817 [anonymous-instance:INFO:vmm/src/ > vstate.rs:97] Guest memory starts at 7faddec00000 > > 2019-01-05T20:21:44.1*24*761979 [anonymous-instance:INFO:api_server/src/ > http_service.rs:565] The synchronous Put request on "/actions" with body > "{n "action_type": "InstanceStart"n }" was executed successfully. Status > code: 204 No Content. > 2019-01-05T20:21:44.*141*417423 [anonymous-instance:INFO:vmm/src/ > lib.rs:1163] Vmm is stopping. > > As you can see above the first part (from "received a synchronous Put" > to "The synchronous Put request on "/actions"") takes 5 ms and seems to > be an overhead of preparing VMM state for the guest based on information > passes in to the calls create instance and setup block device. > > The delta between last line timestamp and previous one is around 17 ms and > includes 10m of OSv boot time below. > > And OSv bootchart: > OSv v0.52.0-24-g3628f560 > Cmdline: --bootchart /hello virtio_mmio.device=4K@0xd0000000:5 > virtio-blk::probe() -> found virtio-mmio device ... > Solaris: NOTICE: Cannot find the pool label for '/dev/vblk0.1' > disk read (real mode): 0.00ms, (+0.00ms) > uncompress lzloader.elf: 0.00ms, (+0.00ms) > TLS initialization: 2.19ms, (+2.19ms) > .init functions: 4.08ms, (+1.89ms) > SMP launched: 5.67ms, (+1.59ms) > VFS initialized: 6.63ms, (+0.96ms) > Network initialized: 7.22ms, (+0.59ms) > pvpanic done: 8.34ms, (+1.12ms) > pci enumerated: 8.34ms, (+0.00ms) > drivers probe: 8.40ms, (+0.06ms) > drivers loaded: 9.73ms, (+1.33ms) > ZFS mounted: 10.73ms, (+1.00ms) > Total time: 10.73ms, (+0.00ms) > Hello from C code > > OSv boottime (per bootchart) seems to vary between 7-11 ms. > > Now firecracker team reports that they can boot Linux in 150ms - from the > "The > API server received a synchronous" point to the start of init process. This > is what say per this > https://github.com/firecracker-microvm/firecracker/blob/master/SPECIFICATION.md > : > " > > - It takes <= 125 ms to go from receiving the Firecracker > InstanceStart API call to the start of the Linux guest user-space > /sbin/init process." > > There is some extra info here - > https://github.com/firecracker-microvm/firecracker/issues/792#issuecomment-449853844 > . > > Now I am not sure we can say that OSv is ten times faster to boot as their > numbers are from running firecracker on i3 bare metal instances which might > be slower or faster than my 5-years old laptop. I have not access to i3 > instance (I think they are pretty expensive to rent) to do proper > measurements. On other hand I could run at some point same Linux > configuration on firecracker on my laptop so we could have apple-to-apple > comparison. > > Also I have not done any performance measurements (for example network > I/O) after OSv boots to see how fast it is in that respect and compare to > running on QEMU. > > All in all as far as boot time firecracker seems to be on par with uKVM or > maybe faster. With firecracker overhead of booting OSv seems to be around > 15-20 ms. > > On Sat, Jan 5, 2019 at 12:03 AM Pekka Enberg <[email protected]> wrote: >> >>> >>> >>> On Sat, Jan 5, 2019 at 9:58 AM Pekka Enberg <[email protected]> wrote: >>> >>>> >>>> >>>> On Sat, Jan 5, 2019 at 9:43 AM Waldek Kozaczuk <[email protected]> >>>> wrote: >>>> >>>>> Indeed I have come across some incompatibility issues. Mostly around >>>>> configuration but nothing very fundamental. You can see in some of my >>>>> comments. >>>>> >>>> >>>> Right, I missed the comments because the diff is somewhat large. ;-) >>>> >>>> I would suggest to do Firecracker support in the following (largely >>>> independent) steps: >>>> >>>> - Add support for modern virtio interfaces. Linux has a split driver >>>> base model here, which we could also follow. I.e., have separate >>>> VirtioModern and VirtioLegacy base classes that encapsulate the >>>> differences. >>>> >>> >>> Btw, this can be also tested with QEMU by passing the >>> "disable-modern=off" and "disable-legacy=on" options to the "-device" >>> option that defines the virtio devices. >>> >>> - Pekka >>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "OSv Development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
