date:20190105

Re: [RFC] Support firecracker

2019-01-05 Thread Waldek Kozaczuk

One more thing - our serial console is dreadfully slow 
(https://github.com/cloudius-systems/osv/issues/921). So I think even 
printing bootchart info slows things by 3-5 ms. Without bootchart the 17 ms 
number drops to 12-11 ms.

On Saturday, January 5, 2019 at 10:32:18 PM UTC-5, Waldek Kozaczuk wrote:
>
>
>
> On Saturday, January 5, 2019 at 4:09:23 AM UTC-5, דור לאור wrote:
>>
>> Great stuff Waldek! Is unix socket the normal way to pass parameters for 
>> Firecracker?
>>
> I think so. There is also experimental vsock interface but I am not 
> familiar with it and not sure what purpose of it would be (
> https://github.com/firecracker-microvm/firecracker/issues/650).  
>
>>
>> How's the boot speed vs Qemu?
>>
> I have posted some numbers in my other email to the group but here are 
> some more details.
>
> First of all my numbers come from running tests on my 5-years old MacBook 
> pro that I have been using for all my OSv development in last three years. 
> It is 4-core 2.3 GHz i7 machine. Not sure how it compares to newer models 
> but given Moore's law does not work any more it might be still pretty fast.
>
> Architecture:x86_64
> CPU op-mode(s):  32-bit, 64-bit
> Byte Order:  Little Endian
> CPU(s):  8
> On-line CPU(s) list: 0-7
> Thread(s) per core:  2
> Core(s) per socket:  4
> Socket(s):   1
> NUMA node(s):1
> Vendor ID:   GenuineIntel
> CPU family:  6
> Model:   70
> Model name:  Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
> Stepping:1
> CPU MHz: 898.037
> CPU max MHz: 3500.
> CPU min MHz: 800.
> BogoMIPS:4589.68
> Virtualization:  VT-x
> L1d cache:   32K
> L1i cache:   32K
> L2 cache:256K
> L3 cache:6144K
> L4 cache:131072K
>
> Additionally firecracker startup/configuration is more fine-grained than 
> QEMU which does everything in one shot (start VMM process, configure 
> resources, start guest, etc). With firecracker it is broken down like 
> follows:
>
>1. Start VMM process which listen on socket for API calls (I have not 
>measured it but seems very fast).
>2. Make API call to:
>   - create instance that specified number of vCPUs, memory and kernel 
>   loader file path
>   - configure block device
>   - configure networking device
>   - all these calls seem to take less than 1ms and can be executed in 
>   any order it seems
>3. Make API call to start instance that eventually starts guest
>
> So here are some log snippets (2,3) and OSv bootchart from running 
> native-example rofs image (no networking):
>
> Add block device:
> 2019-01-05T20:21:44.*096*617431 [anonymous-instance:INFO:api_server/src/
> http_service.rs:599] The API server received a synchronous Put request on 
> "/drives/rootfs" with body "{n "drive_id": "rootfs",n "path_on_host": 
> "/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device": 
> false,n "is_read_only": falsen }".
> 2019-01-05T20:21:44.*096*659174 [anonymous-instance:INFO:api_server/src/
> http_service.rs:565] The synchronous Put request on "/drives/rootfs" with 
> body "{n "drive_id": "rootfs",n "path_on_host": 
> "/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device": 
> false,n "is_read_only": falsen }" was executed successfully. Status code: 
> 204 No Content.
>
> Create instance:
> 2019-01-05T20:21:44.*1085*43339 [anonymous-instance:INFO:api_server/src/
> http_service.rs:599] The API server received a synchronous Put request on 
> "/boot-source" with body "{n "kernel_image_path": 
> "/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n 
> "boot_args": "--bootchart /hello"n }".
> 2019-01-05T20:21:44.*108*584295 [anonymous-instance:INFO:api_server/src/
> http_service.rs:565] The synchronous Put request on "/boot-source" with 
> body "{n "kernel_image_path": 
> "/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n 
> "boot_args": "--bootchart /hello"n }" was executed successfully. Status 
> code: 204 No Content.
>
> Start instance that starts guest and terminates the process eventually:
> 2019-01-05T20:21:44.119820357 [anonymous-instance:INFO:api_server/src/
> http_service.rs:599] The API server received a synchronous Put request on 
> "/actions" with body "{n "action_type": "InstanceStart"n }".
> 2019-01-05T20:21:44.119837722 [anonymous-instance:INFO:vmm/src/lib.rs:1104] 
> VMM received instance start command
> 2019-01-05T20:21:44.119903817 [anonymous-instance:INFO:vmm/src/
> vstate.rs:97] Guest memory starts at 7faddec0
>
> 2019-01-05T20:21:44.1*24*761979 [anonymous-instance:INFO:api_server/src/
> http_service.rs:565] The synchronous Put request on "/actions" with body 
> "{n "action_type": "InstanceStart"n }" was executed successfully. Status 
> code: 204 No Content.
> 2019-01-05T20:21:44.*141*417423 [anonymous-instance:INFO:vmm/src/
> lib.rs:1163]

Re: [RFC] Support firecracker

2019-01-05 Thread Waldek Kozaczuk



On Saturday, January 5, 2019 at 2:58:14 AM UTC-5, Pekka Enberg wrote:
>
>
>
> On Sat, Jan 5, 2019 at 9:43 AM Waldek Kozaczuk  > wrote:
>
>> Indeed I have come across some incompatibility issues. Mostly around 
>> configuration but nothing very fundamental. You can see in some of my 
>> comments. 
>>
>
> Right, I missed the comments because the diff is somewhat large. ;-)
>
> I would suggest to do Firecracker support in the following (largely 
> independent) steps:
>
> - Add support for modern virtio interfaces. Linux has a split driver base 
> model here, which we could also follow. I.e., have separate VirtioModern 
> and VirtioLegacy base classes that encapsulate the differences.
>
> - Add support for VirtIO MMIO. Most of this is restructuring the drivers 
> to use transport-independent interfaces and then have VirtioPCI and 
> VirtioMMIO classes that implement transport-specific logic.
>
> - Add Intel mptable (used by Firecracker) support as an alternative to 
> ACPI hardware discovery.
>
> - Add any misc Firecraker related things that are still needed to make OSv 
> boot and run.
>
> - Pekka
>

I like your suggestion. The extra thing would be adjusting OSv to support 
Linux boot protocol which would be also handy to run it on hyperkit or 
bhyve.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [RFC] Support firecracker

2019-01-05 Thread Waldek Kozaczuk

On Saturday, January 5, 2019 at 4:09:23 AM UTC-5, דור לאור wrote:
>
> Great stuff Waldek! Is unix socket the normal way to pass parameters for 
> Firecracker?
>
I think so. There is also experimental vsock interface but I am not 
familiar with it and not sure what purpose of it would be 
(https://github.com/firecracker-microvm/firecracker/issues/650).  

>
> How's the boot speed vs Qemu?
>
I have posted some numbers in my other email to the group but here are some 
more details.

First of all my numbers come from running tests on my 5-years old MacBook 
pro that I have been using for all my OSv development in last three years. 
It is 4-core 2.3 GHz i7 machine. Not sure how it compares to newer models 
but given Moore's law does not work any more it might be still pretty fast.

Architecture:x86_64
CPU op-mode(s):  32-bit, 64-bit
Byte Order:  Little Endian
CPU(s):  8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):   1
NUMA node(s):1
Vendor ID:   GenuineIntel
CPU family:  6
Model:   70
Model name:  Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
Stepping:1
CPU MHz: 898.037
CPU max MHz: 3500.
CPU min MHz: 800.
BogoMIPS:4589.68
Virtualization:  VT-x
L1d cache:   32K
L1i cache:   32K
L2 cache:256K
L3 cache:6144K
L4 cache:131072K

Additionally firecracker startup/configuration is more fine-grained than 
QEMU which does everything in one shot (start VMM process, configure 
resources, start guest, etc). With firecracker it is broken down like 
follows:

   1. Start VMM process which listen on socket for API calls (I have not 
   measured it but seems very fast).
   2. Make API call to:
  - create instance that specified number of vCPUs, memory and kernel 
  loader file path
  - configure block device
  - configure networking device
  - all these calls seem to take less than 1ms and can be executed in 
  any order it seems
   3. Make API call to start instance that eventually starts guest

So here are some log snippets (2,3) and OSv bootchart from running 
native-example rofs image (no networking):

Add block device:
2019-01-05T20:21:44.*096*617431 
[anonymous-instance:INFO:api_server/src/http_service.rs:599] The API server 
received a synchronous Put request on "/drives/rootfs" with body "{n 
"drive_id": "rootfs",n "path_on_host": 
"/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device": 
false,n "is_read_only": falsen }".
2019-01-05T20:21:44.*096*659174 
[anonymous-instance:INFO:api_server/src/http_service.rs:565] The 
synchronous Put request on "/drives/rootfs" with body "{n "drive_id": 
"rootfs",n "path_on_host": 
"/home/wkozaczuk/projects/osv/build/release/usr.rofs",n "is_root_device": 
false,n "is_read_only": falsen }" was executed successfully. Status code: 
204 No Content.

Create instance:
2019-01-05T20:21:44.*1085*43339 
[anonymous-instance:INFO:api_server/src/http_service.rs:599] The API server 
received a synchronous Put request on "/boot-source" with body "{n 
"kernel_image_path": 
"/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n 
"boot_args": "--bootchart /hello"n }".
2019-01-05T20:21:44.*108*584295 
[anonymous-instance:INFO:api_server/src/http_service.rs:565] The 
synchronous Put request on "/boot-source" with body "{n 
"kernel_image_path": 
"/home/wkozaczuk/projects/waldek-osv/build/release/loader-stripped.elf",n 
"boot_args": "--bootchart /hello"n }" was executed successfully. Status 
code: 204 No Content.

Start instance that starts guest and terminates the process eventually:
2019-01-05T20:21:44.119820357 
[anonymous-instance:INFO:api_server/src/http_service.rs:599] The API server 
received a synchronous Put request on "/actions" with body "{n 
"action_type": "InstanceStart"n }".
2019-01-05T20:21:44.119837722 [anonymous-instance:INFO:vmm/src/lib.rs:1104] 
VMM received instance start command
2019-01-05T20:21:44.119903817 
[anonymous-instance:INFO:vmm/src/vstate.rs:97] Guest memory starts at 
7faddec0

2019-01-05T20:21:44.1*24*761979 
[anonymous-instance:INFO:api_server/src/http_service.rs:565] The 
synchronous Put request on "/actions" with body "{n "action_type": 
"InstanceStart"n }" was executed successfully. Status code: 204 No Content.
2019-01-05T20:21:44.*141*417423 
[anonymous-instance:INFO:vmm/src/lib.rs:1163] Vmm is stopping.

As you can see above the first part  (from "received a synchronous Put" to "The 
synchronous Put request on "/actions"") takes 5 ms and seems to be an 
overhead of preparing VMM state for the guest based on information passes 
in to the calls create instance and setup block device.

The delta between last line timestamp and previous one is around 17 ms and 
includes 10m of OSv boot time below.

And OSv bootchart:
OSv v0.52.0-24-g3628f560
Cmdline: --bootchart /hello

Build failed in Jenkins: osv-build-nightly #1708

2019-01-05 Thread jenkins

See 


--
[...truncated 139.08 KB...]
Adding /usr/lib/jvm/java/jre/lib/zi/America/Lima...
Adding /usr/lib/jvm/java/jre/lib/management/jmxremote.password.template...
Adding /usr/lib/jvm/java/jre/lib/zi/Africa/Ceuta...
Adding /usr/lib/jvm/java/jre/lib/zi/Asia/Kolkata...
Adding /usr/lib/jvm/java/jre/lib/zi/Pacific/Chatham...
Adding /usr/lib/jvm/java/jre/lib/zi/Asia/Thimphu...
Adding /usr/lib/jvm/java/jre/lib/zi/America/Yellowknife...
Adding /usr/lib/jvm/java/jre/lib/zi/Etc/GMT-3...
Adding /usr/lib/jvm/java/jre/lib/zi/Asia/Kuching...
Adding /usr/lib/jvm/java/jre/lib/zi/Africa/Brazzaville...
Adding /usr/lib/jvm/java/jre/lib/zi/Europe/Malta...
Adding /usr/lib/jvm/java/jre/lib/zi/Europe/Bucharest...
Adding /usr/lib/jvm/java/jre/lib/zi/America/Puerto_Rico...
Adding /usr/lib/jvm/java/jre/lib/zi/Asia/Ashgabat...
Adding /usr/lib/jvm/java/jre/lib/zi/America/Fortaleza...
Adding /usr/lib/jvm/java/jre/lib/zi/GMT...
Adding /usr/lib/jvm/java/jre/lib/zi/Asia/Novosibirsk...
Adding /usr/lib/jvm/java/jre/lib/amd64/jli/libjli.so...
Adding /usr/lib/jvm/java/jre/bin/unpack200...
Adding /usr/lib/jvm/java/jre/lib/rhino.jar...
Adding /usr/lib/jvm/java/jre/lib/zi/Asia/Dhaka...
Adding /usr/lib/jvm/java/jre/lib/zi/Pacific/Norfolk...
Adding /usr/lib/jvm/java/jre/lib/amd64/libnet.so...
Adding /usr/lib/jvm/java/jre/lib/zi/Europe/Madrid...
Adding /usr/lib/jvm/java/jre/lib/amd64/libverify.so...
Adding /usr/lib/jvm/java/jre/lib/zi/America/Merida...
Adding /usr/lib/jvm/java/jre/lib/amd64/libhprof.so...
Adding /usr/lib/jvm/java/jre/lib/zi/Australia/Eucla...
Adding /usr/lib/jvm/java/jre/lib/amd64/libj2gss.so...
Adding /usr/lib/jvm/java/jre/lib/amd64/libattach.so...
Adding /usr/lib/jvm/java/jre/lib/zi/Africa/Dar_es_Salaam...
Adding /usr/lib/jvm/java/jre/lib/zi/Asia/Krasnoyarsk...
Adding /usr/lib/jvm/java/jre/lib/zi/Pacific/Wake...
Adding /usr/lib/jvm/java/jre/lib/zi/Pacific/Kiritimati...
Adding /usr/lib/jvm/java/jre/lib/zi/America/Mazatlan...
Adding /usr/lib/jvm/java/jre/lib/zi/Asia/Vientiane...
Adding /usr/lib/jvm/java/jre/lib/zi/Pacific/Enderbury...
Adding /usr/lib/jvm/java/lib/amd64/jli/libjli.so...
Adding /usr/lib/jvm/java/jre/lib/zi/America/Miquelon...
Adding /usr/lib/jvm/java/jre/lib/zi/America/Mexico_City...
Adding /usr/lib/jvm/java/jre/lib/zi/Pacific/Rarotonga...
Adding /usr/lib/jvm/java/jre/lib/zi/Pacific/Tarawa...
Adding /usr/lib/jvm/java/jre/lib/zi/Asia/Novokuznetsk...
Adding /usr/lib/jvm/java/jre/lib/zi/Antarctica/Casey...
Adding /usr/lib/jvm/java/jre/lib/zi/Antarctica/Palmer...
Link /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.60-2.4.7.4.fc20.x86_64 to java ...
Link /usr/lib/jvm/jre to java/jre ...
Link /usr/lib/jvm/java/jre/lib/security/cacerts to /etc/pki/java/cacerts ...
Adding /java_isolated.so...
Adding /java/runjava-isolated.jar...
Adding /.java.policy...
Adding /tests/ClassPutInRoot.class...
Adding /tests/java/isolates.jar...
Adding /usr/lib/jvm/java/jre/lib/ext/tests-jre-extension.jar...
Adding /tests/java/tests.jar...
Adding /usr/lib/libboost_unit_test_framework.so.1.55.0...
Adding /usr/lib/libboost_filesystem.so.1.55.0...
Adding /testrunner.so...
Adding /tests/tst-pthread.so...
Adding /tests/misc-ramdisk.so...
Adding /tests/tst-vblk.so...
Adding /tests/tst-bsd-evh.so...
Adding /tests/misc-bsd-callout.so...
Adding /tests/tst-bsd-kthread.so...
Adding /tests/tst-bsd-taskqueue.so...
Adding /tests/tst-fpu.so...
Adding /tests/tst-preempt.so...
Adding /tests/tst-tracepoint.so...
Adding /tests/tst-hub.so...
Adding /tests/misc-console.so...
Adding /tests/misc-leak.so...
Adding /tests/misc-readbench.so...
Adding /tests/misc-mmap-anon-perf.so...
Adding /tests/tst-mmap-file.so...
Adding /tests/misc-mmap-big-file.so...
Adding /tests/tst-mmap.so...
Adding /tests/tst-huge.so...
Adding /tests/tst-elf-permissions.so...
Adding /tests/misc-mutex.so...
Adding /tests/misc-sockets.so...
Adding /tests/tst-condvar.so...
Adding /tests/tst-queue-mpsc.so...
Adding /tests/tst-af-local.so...
Adding /tests/tst-pipe.so...
Adding /tests/tst-yield.so...
Adding /tests/misc-ctxsw.so...
Adding /tests/tst-read.so...
Adding /tests/tst-symlink.so...
Adding /tests/tst-openat.so...
Adding /tests/tst-eventfd.so...
Adding /tests/tst-remove.so...
Adding /tests/misc-wake.so...
Adding /tests/tst-epoll.so...
Adding /tests/misc-lfring.so...
Adding /tests/misc-fsx.so...
Adding /tests/tst-sleep.so...
Adding /tests/tst-resolve.so...
Adding /tests/tst-except.so...
Adding /tests/misc-tcp-sendonly.so...
Adding /tests/tst-tcp-nbwrite.so...
Adding /tests/misc-tcp-hash-srv.so...
Adding /tests/misc-loadbalance.so...
Adding /tests/misc-scheduler.so...
Adding /tests/tst-console.so...
Adding /tests/tst-app.so...
Adding /tests/misc-setpriority.so...
Adding /tests/misc-timeslice.so...
Adding /tests/misc-tls.so...
Adding /tests/misc-gtod.so...
Adding /tests/tst-dns-resolver.so...
Adding /tests/tst-kill.so...
Adding /tests/tst-truncate.so...
Adding

Re: [RFC] Support firecracker

2019-01-05 Thread Dor Laor

Great stuff Waldek! Is unix socket the normal way to pass parameters for
Firecracker?

How's the boot speed vs Qemu?

On Sat, Jan 5, 2019 at 12:03 AM Pekka Enberg  wrote:

>
>
> On Sat, Jan 5, 2019 at 9:58 AM Pekka Enberg  wrote:
>
>>
>>
>> On Sat, Jan 5, 2019 at 9:43 AM Waldek Kozaczuk 
>> wrote:
>>
>>> Indeed I have come across some incompatibility issues. Mostly around
>>> configuration but nothing very fundamental. You can see in some of my
>>> comments.
>>>
>>
>> Right, I missed the comments because the diff is somewhat large. ;-)
>>
>> I would suggest to do Firecracker support in the following (largely
>> independent) steps:
>>
>> - Add support for modern virtio interfaces. Linux has a split driver base
>> model here, which we could also follow. I.e., have separate VirtioModern
>> and VirtioLegacy base classes that encapsulate the differences.
>>
>
> Btw, this can be also tested with QEMU by passing the "disable-modern=off"
> and "disable-legacy=on" options to the "-device" option that defines the
> virtio devices.
>
> - Pekka
>
>> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [RFC] Support firecracker

2019-01-05 Thread Pekka Enberg

On Sat, Jan 5, 2019 at 9:58 AM Pekka Enberg  wrote:

>
>
> On Sat, Jan 5, 2019 at 9:43 AM Waldek Kozaczuk 
> wrote:
>
>> Indeed I have come across some incompatibility issues. Mostly around
>> configuration but nothing very fundamental. You can see in some of my
>> comments.
>>
>
> Right, I missed the comments because the diff is somewhat large. ;-)
>
> I would suggest to do Firecracker support in the following (largely
> independent) steps:
>
> - Add support for modern virtio interfaces. Linux has a split driver base
> model here, which we could also follow. I.e., have separate VirtioModern
> and VirtioLegacy base classes that encapsulate the differences.
>

Btw, this can be also tested with QEMU by passing the "disable-modern=off"
and "disable-legacy=on" options to the "-device" option that defines the
virtio devices.

- Pekka

>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [RFC] Support firecracker

Re: [RFC] Support firecracker

Re: [RFC] Support firecracker

Build failed in Jenkins: osv-build-nightly #1708

Re: [RFC] Support firecracker

Re: [RFC] Support firecracker

6 matches

Site Navigation

Mail list logo

Footer information