On Sun, Jan 6, 2019 at 11:32 AM Waldek Kozaczuk <jwkozac...@gmail.com> wrote:
> > > On Saturday, January 5, 2019 at 4:09:23 AM UTC-5, דור לאור wrote: >> >> Great stuff Waldek! Is unix socket the normal way to pass parameters for >> Firecracker? >> > I think so. There is also experimental vsock interface but I am not > familiar with it and not sure what purpose of it would be ( > https://github.com/firecracker-microvm/firecracker/issues/650). > Vsock was added for easier guest host communication with zero configuration (I happen to the author of the vsock kernel module). In theory, they can use vsock for anything that a socket is useful between host and guest. Btw, there are some work to implement nfs over vsock. > >> How's the boot speed vs Qemu? >> > I have posted some numbers in my other email to the group but here are > some more details. > > First of all my numbers come from running tests on my 5-years old MacBook > pro that I have been using for all my OSv development in last three years. > It is 4-core 2.3 GHz i7 machine. Not sure how it compares to newer models > but given Moore's law does not work any more it might be still pretty fast. > > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 8 > On-line CPU(s) list: 0-7 > Thread(s) per core: 2 > Core(s) per socket: 4 > Socket(s): 1 > NUMA node(s): 1 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 70 > Model name: Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz > Stepping: 1 > CPU MHz: 898.037 > CPU max MHz: 3500.0000 > CPU min MHz: 800.0000 > BogoMIPS: 4589.68 > Virtualization: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache: 256K > L3 cache: 6144K > L4 cache: 131072K > > Additionally firecracker startup/configuration is more fine-grained than > QEMU which does everything in one shot (start VMM process, configure > resources, start guest, etc). With firecracker it is broken down like > follows: > > 1. Start VMM process which listen on socket for API calls (I have not > measured it but seems very fast). > 2. Make API call to: > - create instance that specified number of vCPUs, memory and kernel > loader file path > - configure block device > - configure networking device > - all these calls seem to take less than 1ms and can be executed in > any order it seems > 3. Make API call to start instance that eventually starts guest > > So here are some log snippets (2,3) and OSv bootchart from running > native-example rofs image (no networking): > > Add block device: > 2019-01-05T20:21:44.*096*617431 [anonymous-instance:INFO:api_server/src/ > http_service.rs:599] The API server received a synchronous Put request on > "/drives/rootfs" with body "{n "drive_id": "rootfs",n "path_on_host": > "/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device": > false,n "is_read_only": falsen }". > 2019-01-05T20:21:44.*096*659174 [anonymous-instance:INFO:api_server/src/ > http_service.rs:565] The synchronous Put request on "/drives/rootfs" with > body "{n "drive_id": "rootfs",n "path_on_host": > "/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device": > false,n "is_read_only": falsen }" was executed successfully. Status code: > 204 No Content. > > Create instance: > 2019-01-05T20:21:44.*1085*43339 [anonymous-instance:INFO:api_server/src/ > http_service.rs:599] The API server received a synchronous Put request on > "/boot-source" with body "{n "kernel_image_path": > "/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n > "boot_args": "--bootchart /hello"n }". > 2019-01-05T20:21:44.*108*584295 [anonymous-instance:INFO:api_server/src/ > http_service.rs:565] The synchronous Put request on "/boot-source" with > body "{n "kernel_image_path": > "/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n > "boot_args": "--bootchart /hello"n }" was executed successfully. Status > code: 204 No Content. > > Start instance that starts guest and terminates the process eventually: > 2019-01-05T20:21:44.119820357 [anonymous-instance:INFO:api_server/src/ > http_service.rs:599] The API server received a synchronous Put request on > "/actions" with body "{n "action_type": "InstanceStart"n }". > 2019-01-05T20:21:44.119837722 [anonymous-instance:INFO:vmm/src/lib.rs:1104] > VMM received instance start command > 2019-01-05T20:21:44.119903817 [anonymous-instance:INFO:vmm/src/ > vstate.rs:97] Guest memory starts at 7faddec00000 > > 2019-01-05T20:21:44.1*24*761979 [anonymous-instance:INFO:api_server/src/ > http_service.rs:565] The synchronous Put request on "/actions" with body > "{n "action_type": "InstanceStart"n }" was executed successfully. Status > code: 204 No Content. > 2019-01-05T20:21:44.*141*417423 [anonymous-instance:INFO:vmm/src/ > lib.rs:1163] Vmm is stopping. > > As you can see above the first part (from "received a synchronous Put" > to "The synchronous Put request on "/actions"") takes 5 ms and seems to > be an overhead of preparing VMM state for the guest based on information > passes in to the calls create instance and setup block device. > > The delta between last line timestamp and previous one is around 17 ms and > includes 10m of OSv boot time below. > > And OSv bootchart: > OSv v0.52.0-24-g3628f560 > Cmdline: --bootchart /hello virtio_mmio.device=4K@0xd0000000:5 > virtio-blk::probe() -> found virtio-mmio device ... > Solaris: NOTICE: Cannot find the pool label for '/dev/vblk0.1' > disk read (real mode): 0.00ms, (+0.00ms) > uncompress lzloader.elf: 0.00ms, (+0.00ms) > TLS initialization: 2.19ms, (+2.19ms) > .init functions: 4.08ms, (+1.89ms) > SMP launched: 5.67ms, (+1.59ms) > VFS initialized: 6.63ms, (+0.96ms) > Network initialized: 7.22ms, (+0.59ms) > pvpanic done: 8.34ms, (+1.12ms) > pci enumerated: 8.34ms, (+0.00ms) > drivers probe: 8.40ms, (+0.06ms) > drivers loaded: 9.73ms, (+1.33ms) > ZFS mounted: 10.73ms, (+1.00ms) > Total time: 10.73ms, (+0.00ms) > Hello from C code > > OSv boottime (per bootchart) seems to vary between 7-11 ms. > > Now firecracker team reports that they can boot Linux in 150ms - from the "The > API server received a synchronous" point to the start of init process. This > is what say per this > https://github.com/firecracker-microvm/firecracker/blob/master/SPECIFICATION.md > : > " > > - It takes <= 125 ms to go from receiving the Firecracker > InstanceStart API call to the start of the Linux guest user-space > /sbin/init process." > > There is some extra info here - > https://github.com/firecracker-microvm/firecracker/issues/792#issuecomment-449853844 > . > > Now I am not sure we can say that OSv is ten times faster to boot as their > numbers are from running firecracker on i3 bare metal instances which might > be slower or faster than my 5-years old laptop. I have not access to i3 > instance (I think they are pretty expensive to rent) to do proper > measurements. On other hand I could run at some point same Linux > configuration on firecracker on my laptop so we could have apple-to-apple > comparison. > > Also I have not done any performance measurements (for example network > I/O) after OSv boots to see how fast it is in that respect and compare to > running on QEMU. > > All in all as far as boot time firecracker seems to be on par with uKVM or > maybe faster. With firecracker overhead of booting OSv seems to be around > 15-20 ms. > > On Sat, Jan 5, 2019 at 12:03 AM Pekka Enberg <pen...@scylladb.com> wrote: >> >>> >>> >>> On Sat, Jan 5, 2019 at 9:58 AM Pekka Enberg <pen...@scylladb.com> wrote: >>> >>>> >>>> >>>> On Sat, Jan 5, 2019 at 9:43 AM Waldek Kozaczuk <jwkoz...@gmail.com> >>>> wrote: >>>> >>>>> Indeed I have come across some incompatibility issues. Mostly around >>>>> configuration but nothing very fundamental. You can see in some of my >>>>> comments. >>>>> >>>> >>>> Right, I missed the comments because the diff is somewhat large. ;-) >>>> >>>> I would suggest to do Firecracker support in the following (largely >>>> independent) steps: >>>> >>>> - Add support for modern virtio interfaces. Linux has a split driver >>>> base model here, which we could also follow. I.e., have separate >>>> VirtioModern and VirtioLegacy base classes that encapsulate the >>>> differences. >>>> >>> >>> Btw, this can be also tested with QEMU by passing the >>> "disable-modern=off" and "disable-legacy=on" options to the "-device" >>> option that defines the virtio devices. >>> >>> - Pekka >>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "OSv Development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to osv-dev+u...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "OSv Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to osv-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Asias -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
