On Saturday, January 5, 2019 at 4:09:23 AM UTC-5, דור לאור wrote:
>
> Great stuff Waldek! Is unix socket the normal way to pass parameters for
> Firecracker?
>
I think so. There is also experimental vsock interface but I am not
familiar with it and not sure what purpose of it would be
(https://github.com/firecracker-microvm/firecracker/issues/650).
>
> How's the boot speed vs Qemu?
>
I have posted some numbers in my other email to the group but here are some
more details.
First of all my numbers come from running tests on my 5-years old MacBook
pro that I have been using for all my OSv development in last three years.
It is 4-core 2.3 GHz i7 machine. Not sure how it compares to newer models
but given Moore's law does not work any more it might be still pretty fast.
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 70
Model name: Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
Stepping: 1
CPU MHz: 898.037
CPU max MHz: 3500.0000
CPU min MHz: 800.0000
BogoMIPS: 4589.68
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
L4 cache: 131072K
Additionally firecracker startup/configuration is more fine-grained than
QEMU which does everything in one shot (start VMM process, configure
resources, start guest, etc). With firecracker it is broken down like
follows:
1. Start VMM process which listen on socket for API calls (I have not
measured it but seems very fast).
2. Make API call to:
- create instance that specified number of vCPUs, memory and kernel
loader file path
- configure block device
- configure networking device
- all these calls seem to take less than 1ms and can be executed in
any order it seems
3. Make API call to start instance that eventually starts guest
So here are some log snippets (2,3) and OSv bootchart from running
native-example rofs image (no networking):
Add block device:
2019-01-05T20:21:44.*096*617431
[anonymous-instance:INFO:api_server/src/http_service.rs:599] The API server
received a synchronous Put request on "/drives/rootfs" with body "{n
"drive_id": "rootfs",n "path_on_host":
"/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device":
false,n "is_read_only": falsen }".
2019-01-05T20:21:44.*096*659174
[anonymous-instance:INFO:api_server/src/http_service.rs:565] The
synchronous Put request on "/drives/rootfs" with body "{n "drive_id":
"rootfs",n "path_on_host":
"/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device":
false,n "is_read_only": falsen }" was executed successfully. Status code:
204 No Content.
Create instance:
2019-01-05T20:21:44.*1085*43339
[anonymous-instance:INFO:api_server/src/http_service.rs:599] The API server
received a synchronous Put request on "/boot-source" with body "{n
"kernel_image_path":
"/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n
"boot_args": "--bootchart /hello"n }".
2019-01-05T20:21:44.*108*584295
[anonymous-instance:INFO:api_server/src/http_service.rs:565] The
synchronous Put request on "/boot-source" with body "{n
"kernel_image_path":
"/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n
"boot_args": "--bootchart /hello"n }" was executed successfully. Status
code: 204 No Content.
Start instance that starts guest and terminates the process eventually:
2019-01-05T20:21:44.119820357
[anonymous-instance:INFO:api_server/src/http_service.rs:599] The API server
received a synchronous Put request on "/actions" with body "{n
"action_type": "InstanceStart"n }".
2019-01-05T20:21:44.119837722 [anonymous-instance:INFO:vmm/src/lib.rs:1104]
VMM received instance start command
2019-01-05T20:21:44.119903817
[anonymous-instance:INFO:vmm/src/vstate.rs:97] Guest memory starts at
7faddec00000
2019-01-05T20:21:44.1*24*761979
[anonymous-instance:INFO:api_server/src/http_service.rs:565] The
synchronous Put request on "/actions" with body "{n "action_type":
"InstanceStart"n }" was executed successfully. Status code: 204 No Content.
2019-01-05T20:21:44.*141*417423
[anonymous-instance:INFO:vmm/src/lib.rs:1163] Vmm is stopping.
As you can see above the first part (from "received a synchronous Put" to "The
synchronous Put request on "/actions"") takes 5 ms and seems to be an
overhead of preparing VMM state for the guest based on information passes
in to the calls create instance and setup block device.
The delta between last line timestamp and previous one is around 17 ms and
includes 10m of OSv boot time below.
And OSv bootchart:
OSv v0.52.0-24-g3628f560
Cmdline: --bootchart /hello virtio_mmio.device=4K@0xd0000000:5
virtio-blk::probe() -> found virtio-mmio device ...
Solaris: NOTICE: Cannot find the pool label for '/dev/vblk0.1'
disk read (real mode): 0.00ms, (+0.00ms)
uncompress lzloader.elf: 0.00ms, (+0.00ms)
TLS initialization: 2.19ms, (+2.19ms)
.init functions: 4.08ms, (+1.89ms)
SMP launched: 5.67ms, (+1.59ms)
VFS initialized: 6.63ms, (+0.96ms)
Network initialized: 7.22ms, (+0.59ms)
pvpanic done: 8.34ms, (+1.12ms)
pci enumerated: 8.34ms, (+0.00ms)
drivers probe: 8.40ms, (+0.06ms)
drivers loaded: 9.73ms, (+1.33ms)
ZFS mounted: 10.73ms, (+1.00ms)
Total time: 10.73ms, (+0.00ms)
Hello from C code
OSv boottime (per bootchart) seems to vary between 7-11 ms.
Now firecracker team reports that they can boot Linux in 150ms - from the "The
API server received a synchronous" point to the start of init process. This
is what say per this
https://github.com/firecracker-microvm/firecracker/blob/master/SPECIFICATION.md:
"
- It takes <= 125 ms to go from receiving the Firecracker InstanceStart
API call to the start of the Linux guest user-space /sbin/init process."
There is some extra info here
-
https://github.com/firecracker-microvm/firecracker/issues/792#issuecomment-449853844.
Now I am not sure we can say that OSv is ten times faster to boot as their
numbers are from running firecracker on i3 bare metal instances which might
be slower or faster than my 5-years old laptop. I have not access to i3
instance (I think they are pretty expensive to rent) to do proper
measurements. On other hand I could run at some point same Linux
configuration on firecracker on my laptop so we could have apple-to-apple
comparison.
Also I have not done any performance measurements (for example network I/O)
after OSv boots to see how fast it is in that respect and compare to
running on QEMU.
All in all as far as boot time firecracker seems to be on par with uKVM or
maybe faster. With firecracker overhead of booting OSv seems to be around
15-20 ms.
On Sat, Jan 5, 2019 at 12:03 AM Pekka Enberg <[email protected]
> <javascript:>> wrote:
>
>>
>>
>> On Sat, Jan 5, 2019 at 9:58 AM Pekka Enberg <[email protected]
>> <javascript:>> wrote:
>>
>>>
>>>
>>> On Sat, Jan 5, 2019 at 9:43 AM Waldek Kozaczuk <[email protected]
>>> <javascript:>> wrote:
>>>
>>>> Indeed I have come across some incompatibility issues. Mostly around
>>>> configuration but nothing very fundamental. You can see in some of my
>>>> comments.
>>>>
>>>
>>> Right, I missed the comments because the diff is somewhat large. ;-)
>>>
>>> I would suggest to do Firecracker support in the following (largely
>>> independent) steps:
>>>
>>> - Add support for modern virtio interfaces. Linux has a split driver
>>> base model here, which we could also follow. I.e., have separate
>>> VirtioModern and VirtioLegacy base classes that encapsulate the differences.
>>>
>>
>> Btw, this can be also tested with QEMU by passing the
>> "disable-modern=off" and "disable-legacy=on" options to the "-device"
>> option that defines the virtio devices.
>>
>> - Pekka
>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "OSv Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
