With the problem size bigger I see OSv consistently beating Linux host (at
least on my laptop, Ubuntu 19.10).
Linux:
./radix -p 1 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
1 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 620025 127815 492206
TIMING INFORMATION
Start time : 1582659397056280
Initialization finish time : 1582659400086786
Overall finish time : 1582659400706811
Total time with initialization : 3650531
Total time without initialization : 620025
./radix -p 2 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
2 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 333298 74005 258944
TIMING INFORMATION
Start time : 1582659471193401
Initialization finish time : 1582659472761435
Overall finish time : 1582659473094733
Total time with initialization : 1901332
Total time without initialization : 333298
./radix -p 4 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
4 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 176586 38013 137823
TIMING INFORMATION
Start time : 1582659586192838
Initialization finish time : 1582659586997985
Overall finish time : 1582659587174571
Total time with initialization : 981733
Total time without initialization : 176586
OSv:
./scripts/run.py -p qemu_microvm --qemu-path
/home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
--nics 0 --nogdb -m 1G -c 1 --block-device-cache writeback,aio=threads -e
'/radix -p 1 -r4096 -n16777216'
OSv v0.54.0-119-g4ee4b788
Booted up in 4.15 ms
Cmdline: /radix -p 1 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
1 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 535304 130142 405145
TIMING INFORMATION
Start time : 1582659265555955
Initialization finish time : 1582659268568276
Overall finish time : 1582659269103580
Total time with initialization : 3547625
Total time without initialization : 535304
./scripts/run.py -p qemu_microvm --qemu-path
/home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
--nics 0 --nogdb -m 1G -c 2 --block-device-cache writeback,aio=threads -e
'/radix -p 2 -r4096 -n16777216'
OSv v0.54.0-119-g4ee4b788
Booted up in 5.39 ms
Cmdline: /radix -p 2 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
2 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 293180 78253 211421
TIMING INFORMATION
Start time : 1582659500041834
Initialization finish time : 1582659501640537
Overall finish time : 1582659501933717
Total time with initialization : 1891883
Total time without initialization : 293180
./scripts/run.py -p qemu_microvm --qemu-path
/home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
--nics 0 --nogdb -m 1G -c 4 --block-device-cache writeback,aio=threads -e
'/radix -p 4 -r4096 -n16777216'
OSv v0.54.0-119-g4ee4b788
Booted up in 5.03 ms
Cmdline: /radix -p 4 -r4096 -n16777216
Integer Radix Sort
16777216 Keys
4 Processors
Radix = 4096
Max key = 524288
PROCESS STATISTICS
Total Rank Sort
Proc Time Time Time
0 163844 47966 114362
TIMING INFORMATION
Start time : 1582659574048566
Initialization finish time : 1582659575031430
Overall finish time : 1582659575195274
Total time with initialization : 1146708
Total time without initialization : 163844
So maybe (at least in this case) OSv scales pretty well with the number of
CPUs:
Most likely because in this use case is very computation-heavy and OSv does
not make many exits to the host (I wonder if OSv tracing mechanism has a
way to show count of all exists).
On Tuesday, February 25, 2020 at 2:12:10 PM UTC-5, [email protected]
wrote:
> Wow, awesome results! I will try to reproduce shortly from here! If you
> want to increase the problem size, add -n 16777216 (which is 64x bigger
> than the current problem size of 262144, something will take take a couple
> of seconds to run).
>
> I am taking note of your commands to run the build and will be doing the
> same on my machine to see what I produce. I will post screenshots :)
>
>
> On Tuesday, February 25, 2020 at 1:09:08 PM UTC-5, Waldek Kozaczuk wrote:
>>
>> So I did try to build and run the radix test (please note my Ubuntu
>> laptop has only 4 cores and hyper-threading disabled). BTW it seems that
>> particular benchmark does not need read-write FS so I used ROFS):
>>
>> ./scripts/manifest_from_host.sh -w ../splash2-posix/kernels/radix/radix
>> && ./scripts/*build* fs=rofs --append-manifest -j4
>>
>> Linux host 1 cpu:
>>
>> ./radix -p 1 -r4096
>>
>>
>> Integer Radix Sort
>>
>> 262144 Keys
>>
>> 1 Processors
>>
>> Radix = 4096
>>
>> Max key = 524288
>>
>>
>>
>> PROCESS STATISTICS
>>
>> Total Rank Sort
>>
>> Proc Time Time Time
>>
>> 0 7335 2568 4765
>>
>>
>> TIMING INFORMATION
>>
>> Start time : 1582652832386234
>>
>> Initialization finish time : 1582652832444092
>>
>> Overall finish time : 1582652832451427
>>
>> Total time with initialization : 65193
>>
>> Total time without initialization : 7335
>>
>>
>> Linux host 2 cpus:
>> ./radix -p 2 -r4096
>>
>> Integer Radix Sort
>> 262144 Keys
>> 2 Processors
>> Radix = 4096
>> Max key = 524288
>>
>>
>> PROCESS STATISTICS
>> Total Rank Sort
>> Proc Time Time Time
>> 0 4325 1571 2704
>>
>> TIMING INFORMATION
>> Start time : 1582652821496771
>> Initialization finish time : 1582652821531279
>> Overall finish time : 1582652821535604
>> Total time with initialization : 38833
>> Total time without initialization : 4325
>>
>> host 4 cpus:
>> ./radix -p 4 -r4096
>>
>> Integer Radix Sort
>> 262144 Keys
>> 4 Processors
>> Radix = 4096
>> Max key = 524288
>>
>>
>> PROCESS STATISTICS
>> Total Rank Sort
>> Proc Time Time Time
>> 0 2599 1077 1470
>>
>> TIMING INFORMATION
>> Start time : 1582653906150199
>> Initialization finish time : 1582653906171932
>> Overall finish time : 1582653906174531
>> Total time with initialization : 24332
>> Total time without initialization : 2599
>>
>>
>> OSv 1 CPU
>> ./scripts/run.py -p qemu_microvm --qemu-path
>> /home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
>>
>> --nics 0 --nogdb -m 64M -c 1 --block-device-cache writeback,aio=threads -e
>> '/radix -p 1 -r4096'
>> OSv v0.54.0-119-g4ee4b788
>> Booted up in 3.75 ms
>> Cmdline: /radix -p 1 -r4096
>>
>> Integer Radix Sort
>> 262144 Keys
>> 1 Processors
>> Radix = 4096
>> Max key = 524288
>>
>>
>> PROCESS STATISTICS
>> Total Rank Sort
>> Proc Time Time Time
>> 0 6060 2002 4049
>>
>> TIMING INFORMATION
>> Start time : 1582652845450708
>> Initialization finish time : 1582652845500348
>> Overall finish time : 1582652845506408
>> Total time with initialization : 55700
>> Total time without initialization : 6060
>>
>> OSv 2 CPUs:
>> ./scripts/run.py -p qemu_microvm --qemu-path
>> /home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
>>
>> --nics 0 --nogdb -m 64M -c 2 --block-device-cache writeback,aio=threads -e
>> '/radix -p 2 -r4096'
>> OSv v0.54.0-119-g4ee4b788
>> Booted up in 4.81 ms
>> Cmdline: /radix -p 2 -r4096
>>
>> Integer Radix Sort
>> 262144 Keys
>> 2 Processors
>> Radix = 4096
>> Max key = 524288
>>
>>
>> PROCESS STATISTICS
>> Total Rank Sort
>> Proc Time Time Time
>> 0 5797 1702 4089
>>
>> TIMING INFORMATION
>> Start time : 1582653305076852
>> Initialization finish time : 1582653305129792
>> Overall finish time : 1582653305135589
>> Total time with initialization : 58737
>> Total time without initialization : 5797
>>
>> OSv 4 cpus
>> ./scripts/run.py -p qemu_microvm --qemu-path
>> /home/wkozaczuk/projects/qemu/bin/release/native/x86_64-softmmu/qemu-system-x86_64
>>
>> --nics 0 --nogdb -m 64M -c 4 --block-device-cache writeback,aio=threads -e
>> '/radix -p 4 -r4096'
>> OSv v0.54.0-119-g4ee4b788
>> Booted up in 5.26 ms
>> Cmdline: /radix -p 4 -r4096
>>
>> Integer Radix Sort
>> 262144 Keys
>> 4 Processors
>> Radix = 4096
>> Max key = 524288
>>
>>
>> PROCESS STATISTICS
>> Total Rank Sort
>> Proc Time Time Time
>> 0 6498 2393 4099
>>
>> TIMING INFORMATION
>> Start time : 1582653946823458
>> Initialization finish time : 1582653946875522
>> Overall finish time : 1582653946882020
>> Total time with initialization : 58562
>> Total time without initialization : 6498
>>
>>
>> As you can see with single CPU the benchmark seems to be 10-15 % faster.
>> But with two and four CPUs OSv barely sees any improvements, whereas on
>> host the app runs 40% faster. So OSv does not seem to scale at all
>> (somebody mentioned it used to) so it would be nice to understand why. OSv
>> has many sophisticated tracing tools that can help here -
>> https://github.com/cloudius-systems/osv/wiki/Trace-analysis-using-trace.py
>>
>> Waldek
>>
>> BTW1. I tried to bump size of the matrix to something higher but with
>> -r8192 the app crashes on both Linux and OSv.
>> BTW2. It would be interestingly to compare OSv with Linux guest (vs host).
>>
>> On Tuesday, February 25, 2020 at 10:05:08 AM UTC-5, [email protected]
>> wrote:
>>>
>>> Thanks for the response! I will get this information to you after work
>>> with the few modifications you recommended! The application is essentially
>>> just testing CPU performance using multiprocessing. Nothing too fancy about
>>> it! The code I am using can be found at:
>>>
>>> https://www.github.com/ProfessorWest/splash2-posix
>>>
>>> In side of the kernels folder located at radix.c and I change the
>>> problem size to 16,777,206.
>>>
>>> If you happen to examine the code, do ignore the lacking cleanness of
>>> the code...we just smashed everything into one file for simplicity on our
>>> end. (Running the same code across all platforms being benchmarked).
>>>
>>> On Tuesday, February 25, 2020 at 8:52:48 AM UTC-5, Waldek Kozaczuk wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am quite late to the party :-) Could you run OSv on single CPU with
>>>> verbose on (add -V to run.py) and send us the output so we can see a
>>>> little
>>>> more what is happening. To disable networking you need to add '--nics=0'
>>>> (for all 50 options run.py supports run it with '--help'). I am not
>>>> familiar with that benchmark but I wonder if it needs read-write FS (ZFS
>>>> in
>>>> OSv case), if not that you can build OSv images with read-only FS
>>>> (./scripts/build fs=rofs). Lastly, you can improve boot time by running
>>>> OSv
>>>> on firecracker (
>>>> https://github.com/cloudius-systems/osv/wiki/Running-OSv-on-Firecracker)
>>>> or on QEMU microvm (-p qemu_imcrovm - requires QEMU >= 4.1), with
>>>> read-only
>>>> FS on both OSv should boot within 5ms, ZFS within 40ms). Last thing -
>>>> writing to console on OSv can be quite slow, I wonder how much this
>>>> benchmark does it.
>>>>
>>>> While I definitely agree with my colleague Nadav, where he essentially
>>>> says do not use OSv if the raw performance matters (database for example)
>>>> and Linux will beat it no matter what, OSv may have advantages in use
>>>> cases
>>>> where pure performance does not matter (it still needs to be reasonable).
>>>> I
>>>> think the best use cases for OSv are serverless or stateless apps
>>>> (microservices or web assembly) running on single CPU where all state
>>>> management is delegated to a remote persistent store (most custom-built
>>>> business apps are like that) and where high isolation matters.
>>>>
>>>> Relatedly, I think it might be more useful to think of OSv (and other
>>>> unikernels) as highly isolated processes. To that end, we still need to
>>>> optimize memory overhead (stacks for example) and improve virtio-fs
>>>> support
>>>> (in this case to run the app on OSv you do not need full image, just
>>>> kernel
>>>> to run a Linux app).
>>>>
>>>> Also, I think the lack of good tooling in unikernel space affects their
>>>> adoption. Compare it with docker - build, push, pull, run. OSv has its
>>>> equivalent - capstan - but at this point, we do not really have a registry
>>>> where one can pull the latest OSv kernel or push, pull images. Trying to
>>>> run an app on OSv is still quite painful to a business app developer - it
>>>> probably takes at least 30 minutes or so.
>>>>
>>>> Lastly, I think one of the main reasons for Docker adoption, was
>>>> repeatability (besides its fantastic ease of use) where one can create an
>>>> image and expect it to run almost the same way in production. Imagine you
>>>> can achieve that with OSv.
>>>>
>>>> Waldek
>>>>
>>>> On Tuesday, February 25, 2020 at 7:00:16 AM UTC-5, [email protected]
>>>> wrote:
>>>>>
>>>>> Very well explained. Thank you for that. That does make perfect sense
>>>>> as well.
>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/4f08a5b2-59ef-453c-aa3c-0f5bcebcfe28%40googlegroups.com.
