Re: [osv-dev] how to monitor OSv's system indicators？

['Dor Laor' via OSv Development]

The easiest is through the host, with KVM, it's just a regular process and
you can use
all standard Linux tools to track it. There are also detailed monitoring
data (forgot where but it's documented)

On Wed, Mar 20, 2024 at 9:49 AM one and  wrote:

> Hello, I have successfully run OSv's native-example application instance
> in Ubuntu, and then I want to monitor OSv's system indicators, such as CPU
> usage, memory usage, network traffic, etc., but I don't know how to do it.
> How to do it, can you help answer it? thanks！！！
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/03f7ada4-187d-4c32-a585-f3ae82f68da1n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAKUaUn4b8ZnQNV6QSRzr%3DszO2H%3DUXXp464qJQK-B9wVcQuHAqA%40mail.gmail.com.

Re: [osv-dev] Virtio-net poor parallel latency

['Dor Laor' via OSv Development]

On Mon, Mar 18, 2024 at 8:29 PM Darren L  wrote:

> Hello!
>
> Thank you for the suggestions. My testing environment is a i9-13900H,
> which has 20 total threads, of which I am allocating the first 8 to OSv and
> the next 4 to the client. These 12 in-use threads exist on 6 hyperthreaded
> cores.
>
> I wasn't sure how to measure the vmexits and/or process scheduling on the
> host.
>

https://access.redhat.com/solutions/6994095
Guest/hypervisor efficiency is many times a function of how many times the
guest
exits to the host. Lower is better


>
> I didn't see netperf on recent versions of OSv in the /tools, it seems to
> have existed in OSv v0.5 then afterwards was removed. I did run a similar
> benchmark on Python 3.10 and ran the same parallel tests, and I received
> much lower latency numbers compared to the Java 8 version. In this case, I
> received latency numbers of 45ms (min-max 18-63, std: 12) compared to a
> possibly confusing measurement for Linux VM of 837ms (min-max 39-1903, std:
> 817). If this does suggest that the JVM is the issue, what steps should I
> take to debug this problem? The application I am using must use Java 8 to
> run; it cannot be run on any other platform.
>

Eliminate Java is one option. Another is to use a recent JVM (17) and ZGC
and hopefully there wouldn't
be GC events (not sure it's a real issue here)


>
> I know these are not the exact details you requested, but I am more than
> happy to learn how I can capture the other details, if necessary. Thank you!
>
> On Wednesday, March 13, 2024 at 5:23:53 PM UTC-4 דור לאור wrote:
>
>> Lots of good details. It's not simple to figure out what's the issue
>> since
>> you have hypervisor, host, OS and JVM variables.
>>
>> How many threads does the host have? Make sure there are enough hardware
>> threads for the
>> guest, virtio on the host and the client. This way all
>> OSv's runable threads will be schedulable.
>> You can also measure the amount of vmexits and process scheduling on the
>> host.
>> There is a chance the JVM is an issue too, can you do the same with
>> netperf?
>>
>>
>> On Wed, Mar 13, 2024 at 9:14 PM Darren L  wrote:
>>
>>> Hello!
>>>
>>> I was wondering if I could get any pointers on why I am receiving
>>> significant latency issues using the virtio-net driver when processing
>>> multiple parallel clients. Hopefully I can explain my issue enough to be
>>> replicated.
>>>
>>> *Testing environment:*
>>> - Comparison: Ubuntu Server (Linux) VM and OSv (used the option "-nv" in
>>> the run.py script for tap networking)
>>> - In common: 4 CPU cores, 4GB of RAM, QEMU KVM, used "taskset" to pin to
>>> the same cores
>>> - Program: *java-httpserver* program from the apps directory, java8
>>> - What was sent: data of varying sizes (1KB to 1MB, 4MB, 8MB...) on the
>>> same machine to the VMs
>>>
>>> *Observations:*
>>> - With single-threaded requests and low data sizes, I was able to
>>> measure a latency on OSv that is lower than the Linux VM latency
>>> - example: for 32KB I measured ~4ms for OSv and 9.8ms for the Linux
>>> VM
>>> - At high data sizes (256KB+), OSv started to measure a higher latency
>>> than the Linux VM
>>> - When I sent *multiple requests* at the same time, OSv suffered a much
>>> larger average latency penalty
>>> - example, at 1MB data size and 16 parallel requests, average
>>> latency was:
>>> - OSv: 120ms (min-max 14-225ms, std: 62ms)
>>> - Linux VM: 82ms (min-max 24-144ms, std: 34ms) for the Linux VM
>>>
>>> *Other notes:*
>>> - I've been using the OSv profiling tools and have seen that the hot
>>> spots typically were in virtio::virtio_driver::wait_for_queue and
>>> virtio::net::receiver, but I was unable to identify the exact issue on why
>>> this latency is the case
>>> - I also noticed when tracing the network layer (
>>> https://github.com/cloudius-systems/osv/wiki/Trace-analysis-using-trace.py#tracing-network-layer),
>>> there were a lot of *net_packet_handling* lines; about as much as there
>>> were *net_packet_in* lines for 1MB, which might indicate that the
>>> packets are not being processed fast enough and are delayed because it is
>>> put in a queue?
>>>
>>> Hope this is clear enough! I am hoping to understand whether I am
>>> misconfiguring OSv or something similar to figure out why this latency
>>> difference is occurring. Thank you for the help in advance, and happy to
>>> provide any more information as needed.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "OSv Development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to osv-dev+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/osv-dev/0ace3980-5036-4df9-9e46-7396bb20ce9fn%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received 

Re: [osv-dev] Virtio-net poor parallel latency

['Dor Laor' via OSv Development]

Lots of good details. It's not simple to figure out what's the issue since
you have hypervisor, host, OS and JVM variables.

How many threads does the host have? Make sure there are enough hardware
threads for the
guest, virtio on the host and the client. This way all
OSv's runable threads will be schedulable.
You can also measure the amount of vmexits and process scheduling on the
host.
There is a chance the JVM is an issue too, can you do the same with netperf?


On Wed, Mar 13, 2024 at 9:14 PM Darren L  wrote:

> Hello!
>
> I was wondering if I could get any pointers on why I am receiving
> significant latency issues using the virtio-net driver when processing
> multiple parallel clients. Hopefully I can explain my issue enough to be
> replicated.
>
> *Testing environment:*
> - Comparison: Ubuntu Server (Linux) VM and OSv (used the option "-nv" in
> the run.py script for tap networking)
> - In common: 4 CPU cores, 4GB of RAM, QEMU KVM, used "taskset" to pin to
> the same cores
> - Program: *java-httpserver* program from the apps directory, java8
> - What was sent: data of varying sizes (1KB to 1MB, 4MB, 8MB...) on the
> same machine to the VMs
>
> *Observations:*
> - With single-threaded requests and low data sizes, I was able to measure
> a latency on OSv that is lower than the Linux VM latency
> - example: for 32KB I measured ~4ms for OSv and 9.8ms for the Linux VM
> - At high data sizes (256KB+), OSv started to measure a higher latency
> than the Linux VM
> - When I sent *multiple requests* at the same time, OSv suffered a much
> larger average latency penalty
> - example, at 1MB data size and 16 parallel requests, average latency
> was:
> - OSv: 120ms (min-max 14-225ms, std: 62ms)
> - Linux VM: 82ms (min-max 24-144ms, std: 34ms) for the Linux VM
>
> *Other notes:*
> - I've been using the OSv profiling tools and have seen that the hot spots
> typically were in virtio::virtio_driver::wait_for_queue and
> virtio::net::receiver, but I was unable to identify the exact issue on why
> this latency is the case
> - I also noticed when tracing the network layer (
> https://github.com/cloudius-systems/osv/wiki/Trace-analysis-using-trace.py#tracing-network-layer),
> there were a lot of *net_packet_handling* lines; about as much as there
> were *net_packet_in* lines for 1MB, which might indicate that the packets
> are not being processed fast enough and are delayed because it is put in a
> queue?
>
> Hope this is clear enough! I am hoping to understand whether I am
> misconfiguring OSv or something similar to figure out why this latency
> difference is occurring. Thank you for the help in advance, and happy to
> provide any more information as needed.
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/0ace3980-5036-4df9-9e46-7396bb20ce9fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAKUaUn7hAB6AqhXJ4ozOU-pHRdFTobfH3QFXdaUcz395C15yHw%40mail.gmail.com.

Re: [osv-dev] FOSDEM24 Presentation

['Dor Laor' via OSv Development]

Waldek, do you have some standard Linux apps you tested that can be
just loaded to OSv? Will be nice to showcase OSv values, with either
performance or VM convenience when you take an app and quickly boot
it with OSv and it just works.


On Thu, Feb 1, 2024 at 3:11 PM Waldek Kozaczuk  wrote:

> If you happen to be at #FOSDEM
>  in Brussels this
> coming weekend and are interested in recent improvements to the OSv ,
> please come to watch my presentation -
> https://fosdem.org/2024/schedule/event/fosdem-2024-3483-support-dynamically-linked-executables-via-linux-ld-so-and-implement-ena-driver-to-expand-application-of-osv/
>
> "From the beginning, OSv has been designed to implement a subset of the
> Linux flavor of POSIX API. Initially, most Linux applications had to be
> re-compiled from source as shared libraries. Starting with the release
> 0.54.0, it has become possible to run unmodified Linux position-independent
> executables (so-called "PIEs") and position-dependent executables "as-is".
>
> In my presentation, I will talk about how OSv has been enhanced to support
> running statically linked executables. Not only that, the same improvements
> make it possible to execute dynamically linked executables via Linux
> dynamic linker. I will also briefly describe a newly added strace-like
> capability aimed to help analyze system calls on the fly.
>
> In addition, I will explain how we have implemented the ENA networking
> driver and thus made it possible to run OSv on AWS Nitro EC2 instances.
> Finally, I will briefly cover the upcoming build configuration tool, based
> on Xconfig, aimed to let the user select OSv kernel components to be
> included or excluded and various parameters to be configured."
>
> Cheers,
> Waldek
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/877de531-b8f4-4680-a065-4965528951a2n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAKUaUn7KSrxu17hvpN4Pqda4ATi21JSpLk8rcT662LxDxBpP%2BQ%40mail.gmail.com.

Re: [osv-dev] ENA driver

['Dor Laor' via OSv Development]

Sweat!

On Thu, Nov 30, 2023 at 7:52 AM Waldek Kozaczuk 
wrote:

> Hi,
>
> I have just created a pull request that implements OSv ENA driver by
> porting the FreeBSD version -
> https://github.com/cloudius-systems/osv/pull/1283. If you have bandwidth
> and know-how please feel free to review this PR.
>
> This is enough to run OSv on a nitro instance like t3 nano with ramfs
> image (ideally we would like NVMe driver). I have only tested it with
> simple golang httpserver and OSv http monitoring module and only on t3
> nano. It seems to be functional and stable but I still need to run more
> tests to get a better feel.
>
> Regards,
> Waldek
> [image: i-060143e9381af3d55.jpg]
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/32a2dfa7-9407-4a1a-a4fc-5711401dfec8n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAKUaUn4QtA%3DmD97i%3Degw5OSaMHqP6T9GUq4-px%3D7Ahvyi%2BkLAA%40mail.gmail.com.

Re: [osv-dev] OSv and Genezio presentations

['Dor Laor' via OSv Development]

On Tue, Aug 8, 2023 at 10:32 PM Waldek Kozaczuk 
wrote:

> Hi,
>
> I am back after a family summer break vacation so you should hear from me
> more often hopefully.
>
> I would also like to share links to the OSv presentation I gave a month
> ago on the Unikernel Alliance forum:
> - recording:
> https://www.youtube.com/watch?v=KnzRnKTCmjc=PLnqfImRjARAjX5veufPxCkGYntfsl8Eb_=3
> - slides PDF:
> https://drive.google.com/file/d/1XPzjLPRnq58HzW84yn98Ut7GVbOp6_cq/view?pli=1
> - slides on Google docs:
> https://docs.google.com/presentation/d/1-dlart1tFslsdjZVajkKpbyREN0UKGoIo55HvZtAl9s/edit?usp=sharing
>
> I would also like to share the presentation by the Genezio team that has
> been building their serverless platform based on OSv:
>
> - recording:
> https://www.youtube.com/watch?v=xRmRXCY_h6I=PLnqfImRjARAjX5veufPxCkGYntfsl8Eb_=4
> - slides PDF:
> https://drive.google.com/file/d/1G_41wvszW8WAgOW1m108_cAP1SZ5lkOC/view?usp=drive_link
>

Nice to see a real life use case !


>
> Enjoy,
> Waldek Kozaczuk
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/853de34d-67aa-470b-96c1-16f6e8fb7589n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAKUaUn7LFYdRg6d%2Bj6%2BWbCXJBhi9Cnc0h8Na0HEkQJ80i2COOw%40mail.gmail.com.

Re: [osv-dev] OSv can run statically linked executable

['Dor Laor' via OSv Development]

Very impressive, it improves OSv's ability to be a very safe and
fast sandbox

On Mon, Apr 24, 2023 at 6:26 AM Waldek Kozaczuk 
wrote:

> Hi,
>
> Over the recent week, I have been working to get OSv to run a simple
> "hello world" app (aka native-example) built as a position-dependent
> statically linked executable. In essence, I picked up where Pekka Enberg
> left over 8 years ago (see
> https://github.com/cloudius-systems/osv/tree/static-elf). Obviously,
> given these days OSv has pretty robust support of over 70 syscalls (and 60
> more that should be trivial to add), and the remaining work is much more
> manageable.
>
> ./scripts/firecracker.py -e /hello
> OSv v0.57.0-37-g0de155a4
> Booted up in 5.23 ms
> Cmdline: /hello
>  -> syscall: 107
>  -> syscall: 102
>  -> syscall: 108
>  -> syscall: 104
>  -> syscall: 158
>  -> syscall: 012
>  -> syscall: 012
>  -> syscall: 158
>  -> syscall: 218
>  -> syscall: 273
>  -> syscall: 063
>  -> syscall: 302
>  -> syscall: 089
>  -> syscall: 318
>  -> syscall: 228
>  -> syscall: 228
>  -> syscall: 012
>  -> syscall: 012
>  -> syscall: 010
>  -> syscall: 262
>  -> syscall: 016
>  -> syscall: 001
> Hello from C code
>  -> syscall: 231
>
> I will be sending a series of proper patches later after I clean some
> issues but in essence here is a list of things I had to do including what
> Pekka started:
>
>1. Tweak dynamic linker to support static executable:
>   - Handle missing DT_SYMTAB, DT_STRTAB and DT_NEEDED.
>   - Handle ET_EXEC
>   - Support statically-linked executable base address
>2. Add basic handling of static ELF entry point and initial stack
>state setup (see figure 3.9 ("Initial Process Stack") of the x86-64 ABI
>specification - https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf
>   - make sure the RDX registers in zeroed and the basic AUX vector
>   with AT_RANDOM is set up
>   - more is left to do to fully support argv and full aux vector
>3. Add support of the brk() syscall (see issue 1138)
>4. Add dummy support of sys_set_robust_list and set_tid_address
>syscalls (possibly needs something more for multithreaded apps).
>5. Support the arch_prctl syscall that sets the app TLS
>   - this was by far the most complicated element that required
>   changing OSv to store new per-pcpu data pointer in GS register and
>   enhancing both syscall handler and interrupt/page fault handler to 
> detect
>   and switch if needed the FS base to the kernel TLS on entry and back to 
> the
>   app one on exit (see
>   
> https://github.com/cloudius-systems/osv/issues/1137#issuecomment-1512315880
>   )
>6. Fixing a potential bug in handling TCGETS in the console driver.
>7. Implement sys_prlimit
>8. Enable the readlink, geteuid and getegid
>
> This was enough to run a single-threaded app but we will need to implement
> the clone syscall to support multi-threaded apps. In addition, we would
> want to support the static pies as well which I hope should not be very
> difficult.
>
> Regards,
> Waldek
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/3a561595-e072-4980-8375-3b742717dd6dn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAKUaUn4WwpOOrYq9-9CF5M4LCzOTqXWhPhWqPunnpdfu2uVZRQ%40mail.gmail.com.

Re: [osv-dev] OSv talk

['Dor Laor' via OSv Development]

On Thu, Feb 9, 2023 at 5:49 AM Waldek Kozaczuk  wrote:

> The video of the talk has been just released and you can watch it here -
> https://fosdem.org/2023/schedule/event/osvevolution/ (the play button
> should play it). There are also slides attached on that page below.
>
> If you want to see the version of the slides with presenter notes, here
> you go -
> https://docs.google.com/presentation/d/1xcKLI-txmS0_T9QUgo6msRaOGzreU22x-Gzn9beh46I/edit?usp=sharing.
> At the end of the talk I ran out of time so that way you can read
> everything I was supposed to say there.
>
> I hope you will enjoy my talk if you have not watched it live.
>
> Nadav,
> You are right! We might want to jump to a version like 1.0 - maybe it is
> finally time for it anyway.
>

I agree. OSv celebrates 10 years this 2023.
Well done Waldek! Outstanding work and great presentation.


>
> Waldek
>
> On Sunday, February 5, 2023 at 8:24:30 AM UTC-5 n...@scylladb.com wrote:
> On Sun, Feb 5, 2023 at 10:23 AM Nadav Har'El  wrote:
>
> On Sun, Feb 5, 2023 at 9:56 AM Waldek Kozaczuk  wrote:
> If you are going to happen to have some time today from 12AM - 1PM
> Brussels time, please watch my talk here at
> https://fosdem.org/2023/schedule/event/osvevolution/. It should be
> streamable live.
>
> Regards,
> Waldemar Kozaczuk
>
> Good luck!
>
> I'll try to watch live, I'm sure it will be interesting.
>
> Indeed it was interesting!
> Version "0.57"  was for sure a bigger accomplishment than the version
> number suggests.
> And thanks for mentioning me :-)
>
> I hope they will have the recording available somewhere later, in case
> someone missed the live feed.
>
> Nadav.
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/4ec729c1-9a4d-40dd-a27b-a2132ecaf31cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAKUaUn6v2j5X3wPz61z66-NFwoUG-F08zDXH-qHn7-h4QWiq5Q%40mail.gmail.com.

Re: [osv-dev] Evolution of OSv: Towards Greater Modularity and Composability

['Dor Laor' via OSv Development]

Highly recommended, cheers Waldek!

On Fri, Dec 30, 2022 at 3:22 PM Waldek Kozaczuk 
wrote:

> Hi,
>
> If you happen to be in Brussels at the beginning of February, I will be
> giving a talk at FOSDEM 2023 about recent improvements to OSv. Feel free to
> stop by to listen or watch it after:
>
> https://fosdem.org/2023/schedule/event/osvevolution/
>
> Regards,
> Waldek
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/3158f9c9-8f47-4ee6-bf83-3162cf850d3cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAKUaUn5NNH053jhd8SbibG8GSJfak-nJNrUL31Uvgx%2BKTe79cQ%40mail.gmail.com.