On Sun, Jan 6, 2019 at 5:49 PM Asias He <[email protected]> wrote:
>
>
> On Sun, Jan 6, 2019 at 11:32 AM Waldek Kozaczuk <[email protected]>
> wrote:
>
>>
>>
>> On Saturday, January 5, 2019 at 4:09:23 AM UTC-5, דור לאור wrote:
>>>
>>> Great stuff Waldek! Is unix socket the normal way to pass parameters for
>>> Firecracker?
>>>
>> I think so. There is also experimental vsock interface but I am not
>> familiar with it and not sure what purpose of it would be (
>> https://github.com/firecracker-microvm/firecracker/issues/650).
>>
>
> Vsock was added for easier guest host communication with zero
> configuration (I happen to the author of the vsock kernel module). In
> theory, they can use vsock for anything that a socket is useful between
> host and guest.
> Btw, there are some work to implement nfs over vsock.
>
Good point, this way they can preboot OSv and run the app with the vsock
control (although any guest can
be controlled this way too)
>
>
>>
>>> How's the boot speed vs Qemu?
>>>
>> I have posted some numbers in my other email to the group but here are
>> some more details.
>>
>> First of all my numbers come from running tests on my 5-years old MacBook
>> pro that I have been using for all my OSv development in last three years.
>> It is 4-core 2.3 GHz i7 machine. Not sure how it compares to newer models
>> but given Moore's law does not work any more it might be still pretty fast.
>>
>> Architecture: x86_64
>> CPU op-mode(s): 32-bit, 64-bit
>> Byte Order: Little Endian
>> CPU(s): 8
>> On-line CPU(s) list: 0-7
>> Thread(s) per core: 2
>> Core(s) per socket: 4
>> Socket(s): 1
>> NUMA node(s): 1
>> Vendor ID: GenuineIntel
>> CPU family: 6
>> Model: 70
>> Model name: Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
>> Stepping: 1
>> CPU MHz: 898.037
>> CPU max MHz: 3500.0000
>> CPU min MHz: 800.0000
>> BogoMIPS: 4589.68
>> Virtualization: VT-x
>> L1d cache: 32K
>> L1i cache: 32K
>> L2 cache: 256K
>> L3 cache: 6144K
>> L4 cache: 131072K
>>
>> Additionally firecracker startup/configuration is more fine-grained than
>> QEMU which does everything in one shot (start VMM process, configure
>> resources, start guest, etc). With firecracker it is broken down like
>> follows:
>>
>> 1. Start VMM process which listen on socket for API calls (I have not
>> measured it but seems very fast).
>> 2. Make API call to:
>> - create instance that specified number of vCPUs, memory and
>> kernel loader file path
>> - configure block device
>> - configure networking device
>> - all these calls seem to take less than 1ms and can be executed
>> in any order it seems
>> 3. Make API call to start instance that eventually starts guest
>>
>> So here are some log snippets (2,3) and OSv bootchart from running
>> native-example rofs image (no networking):
>>
>> Add block device:
>> 2019-01-05T20:21:44.*096*617431 [anonymous-instance:INFO:api_server/src/
>> http_service.rs:599] The API server received a synchronous Put request
>> on "/drives/rootfs" with body "{n "drive_id": "rootfs",n "path_on_host":
>> "/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device":
>> false,n "is_read_only": falsen }".
>> 2019-01-05T20:21:44.*096*659174 [anonymous-instance:INFO:api_server/src/
>> http_service.rs:565] The synchronous Put request on "/drives/rootfs"
>> with body "{n "drive_id": "rootfs",n "path_on_host":
>> "/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device":
>> false,n "is_read_only": falsen }" was executed successfully. Status code:
>> 204 No Content.
>>
>> Create instance:
>> 2019-01-05T20:21:44.*1085*43339 [anonymous-instance:INFO:api_server/src/
>> http_service.rs:599] The API server received a synchronous Put request
>> on "/boot-source" with body "{n "kernel_image_path":
>> "/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n
>> "boot_args": "--bootchart /hello"n }".
>> 2019-01-05T20:21:44.*108*584295 [anonymous-instance:INFO:api_server/src/
>> http_service.rs:565] The synchronous Put request on "/boot-source" with
>> body "{n "kernel_image_path":
>> "/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n
>> "boot_args": "--bootchart /hello"n }" was executed successfully. Status
>> code: 204 No Content.
>>
>> Start instance that starts guest and terminates the process eventually:
>> 2019-01-05T20:21:44.119820357 [anonymous-instance:INFO:api_server/src/
>> http_service.rs:599] The API server received a synchronous Put request
>> on "/actions" with body "{n "action_type": "InstanceStart"n }".
>> 2019-01-05T20:21:44.119837722 [anonymous-instance:INFO:vmm/src/
>> lib.rs:1104] VMM received instance start command
>> 2019-01-05T20:21:44.119903817 [anonymous-instance:INFO:vmm/src/
>> vstate.rs:97] Guest memory starts at 7faddec00000
>>
>> 2019-01-05T20:21:44.1*24*761979 [anonymous-instance:INFO:api_server/src/
>> http_service.rs:565] The synchronous Put request on "/actions" with body
>> "{n "action_type": "InstanceStart"n }" was executed successfully. Status
>> code: 204 No Content.
>> 2019-01-05T20:21:44.*141*417423 [anonymous-instance:INFO:vmm/src/
>> lib.rs:1163] Vmm is stopping.
>>
>> As you can see above the first part (from "received a synchronous Put"
>> to "The synchronous Put request on "/actions"") takes 5 ms and seems to
>> be an overhead of preparing VMM state for the guest based on information
>> passes in to the calls create instance and setup block device.
>>
>> The delta between last line timestamp and previous one is around 17 ms
>> and includes 10m of OSv boot time below.
>>
>> And OSv bootchart:
>> OSv v0.52.0-24-g3628f560
>> Cmdline: --bootchart /hello virtio_mmio.device=4K@0xd0000000:5
>> virtio-blk::probe() -> found virtio-mmio device ...
>> Solaris: NOTICE: Cannot find the pool label for '/dev/vblk0.1'
>> disk read (real mode): 0.00ms, (+0.00ms)
>> uncompress lzloader.elf: 0.00ms, (+0.00ms)
>> TLS initialization: 2.19ms, (+2.19ms)
>> .init functions: 4.08ms, (+1.89ms)
>> SMP launched: 5.67ms, (+1.59ms)
>> VFS initialized: 6.63ms, (+0.96ms)
>> Network initialized: 7.22ms, (+0.59ms)
>> pvpanic done: 8.34ms, (+1.12ms)
>> pci enumerated: 8.34ms, (+0.00ms)
>> drivers probe: 8.40ms, (+0.06ms)
>> drivers loaded: 9.73ms, (+1.33ms)
>> ZFS mounted: 10.73ms, (+1.00ms)
>> Total time: 10.73ms, (+0.00ms)
>> Hello from C code
>>
>> OSv boottime (per bootchart) seems to vary between 7-11 ms.
>>
>> Now firecracker team reports that they can boot Linux in 150ms - from the
>> "The API server received a synchronous" point to the start of init
>> process. This is what say per this
>> https://github.com/firecracker-microvm/firecracker/blob/master/SPECIFICATION.md
>> :
>> "
>>
>> - It takes <= 125 ms to go from receiving the Firecracker
>> InstanceStart API call to the start of the Linux guest user-space
>> /sbin/init process."
>>
>> There is some extra info here -
>> https://github.com/firecracker-microvm/firecracker/issues/792#issuecomment-449853844
>> .
>>
>> Now I am not sure we can say that OSv is ten times faster to boot as
>> their numbers are from running firecracker on i3 bare metal instances which
>> might be slower or faster than my 5-years old laptop. I have not access to
>> i3 instance (I think they are pretty expensive to rent) to do proper
>> measurements. On other hand I could run at some point same Linux
>> configuration on firecracker on my laptop so we could have apple-to-apple
>> comparison.
>>
>> Also I have not done any performance measurements (for example network
>> I/O) after OSv boots to see how fast it is in that respect and compare to
>> running on QEMU.
>>
>> All in all as far as boot time firecracker seems to be on par with uKVM
>> or maybe faster. With firecracker overhead of booting OSv seems to be
>> around 15-20 ms.
>>
>> On Sat, Jan 5, 2019 at 12:03 AM Pekka Enberg <[email protected]> wrote:
>>>
>>>>
>>>>
>>>> On Sat, Jan 5, 2019 at 9:58 AM Pekka Enberg <[email protected]>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Sat, Jan 5, 2019 at 9:43 AM Waldek Kozaczuk <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Indeed I have come across some incompatibility issues. Mostly around
>>>>>> configuration but nothing very fundamental. You can see in some of my
>>>>>> comments.
>>>>>>
>>>>>
>>>>> Right, I missed the comments because the diff is somewhat large. ;-)
>>>>>
>>>>> I would suggest to do Firecracker support in the following (largely
>>>>> independent) steps:
>>>>>
>>>>> - Add support for modern virtio interfaces. Linux has a split driver
>>>>> base model here, which we could also follow. I.e., have separate
>>>>> VirtioModern and VirtioLegacy base classes that encapsulate the
>>>>> differences.
>>>>>
>>>>
>>>> Btw, this can be also tested with QEMU by passing the
>>>> "disable-modern=off" and "disable-legacy=on" options to the "-device"
>>>> option that defines the virtio devices.
>>>>
>>>> - Pekka
>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "OSv Development" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "OSv Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Asias
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
