Re: [osv-dev] [PATCH] aarch64: mask interrupt and disable timer in the handler

[Pekka Enberg]

On Fri, Mar 19, 2021 at 11:50 PM Waldemar Kozaczuk 
wrote:

> The AArch64 Programmer's Guides Generic Timer (chapter 3.4, page 10)
> states following:
>
> "The interrupts generated by the timer behave in a level-sensitive manner.
>  This means that, once the timer firing condition is reached,
>  the timer will continue to signal an interrupt until one of the following
> situations occurs:
>  - IMASK is set to one, which masks the interrupt.
>  - ENABLE is cleared to 0, which disables the timer.
>  - TVAL or CVAL is written, so that firing condition is no longer met."
>
> This indicates that the timer interrupt handler needs to mask
> or disable the timer, otherwise the interrupt will be re-delivered
> and in some scenarios might cause underlying VMM (like QEMU in TCG mode)
> stop delivering subsequent interrupts. In any case per the description
> from the guide above, it is correct to mask the interrupt and/or disable
> the timer in the timer interrupt handler. This is also what Linux does -
>
> https://github.com/torvalds/linux/blob/edd7ab76847442e299af64a761febd180d71f98d/drivers/clocksource/arm_arch_timer.c#L638-L652
> .
>

Reviewed-by: Pekka Enberg 


>
> Besides the above, the patch also sligthly optimizes the method set()
> to make it set new timer only if calculated number of ticks (tval)
> is greater than 0.
>

Why is this optimization useful/needed?

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND8pH8WJ8PXLMVpjtcNdUOvTbfEyYk-urmJ%3DEfMpoGpNrg%40mail.gmail.com.

Re: [osv-dev] Contributing to osv.io

[Pekka Enberg]

Hi Waldek,

On Thu, Feb 18, 2021 at 1:45 AM Waldek Kozaczuk 
wrote:

> You mean osv from cloudious-systems to osv-io, or vice versa? Or create a
> new organization? How easy is it to move the repos?


Moving repositories is easy between organizations and Github preserves old
URLs too. I would move everything to "osv-io", but that's for someone else
to decide.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND8zR0_4Nm6ju_FbTCfFgYTqvYh%3Dim6VqZTBJXzemRnAMQ%40mail.gmail.com.

Re: [osv-dev] Contributing to osv.io

[Pekka Enberg]

Hi,

On Fri, Feb 12, 2021 at 12:08 AM Matthew Kenigsberg <
matthewkenigsb...@gmail.com> wrote:

> Couldn't find that repo, I was looking in cloudius-systems 臘‍♂️
>

Slightly off-topic: I think it would be good to (eventually) consolidate
the Github organizations. Perhaps move "osv.git" (and dependencies) under
the "OSv" organization.

Regards,

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND--ASvoi7%3DaejEJsO0JYrV8Eq8%2Bytsd01Q2kzEr9q%2BE4w%40mail.gmail.com.

Re: [osv-dev] [PATCH] Reject executables with non-matching machine type

[Pekka Enberg]

Hi Samuel,

On Mon, Nov 30, 2020 at 9:16 PM Samuel Laberge 
wrote:

> I totally agree that would be simpler, I was just trying to match the
> other conditionals around that one. But if it's better as != then I'd be
> happy to change it!


Up to you, really, but yeah "!=" is my preference.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND9HsUV%2BL2ftdb%3DWsb_S4dwAi%3Dra6vq_6p%3DVcf7%2B5f%2Bi1w%40mail.gmail.com.

Re: [osv-dev] [PATCH] Reject executables with non-matching machine type

[Pekka Enberg]

On Sun, Nov 29, 2020 at 11:22 AM Samuel Laberge 
wrote:

> Whenever an ELF file is loaded, its machine identifier is now
> compared to the kernel's. If they differ, the executable is
> rejected.
>
> Fixes #1104
>
> Signed-off-by: Samuel Laberge 
>

Reviewed-by: Pekka Enberg 


> ---
>  arch/aarch64/arch.hh | 2 ++
>  arch/x64/arch.hh | 2 ++
>  core/elf.cc  | 3 +++
>  3 files changed, 7 insertions(+)
>
> diff --git a/arch/aarch64/arch.hh b/arch/aarch64/arch.hh
> index 855f1987..3634a796 100644
> --- a/arch/aarch64/arch.hh
> +++ b/arch/aarch64/arch.hh
> @@ -20,6 +20,8 @@ namespace arch {
>  #define INSTR_SIZE_MIN 4
>  #define ELF_IMAGE_START (OSV_KERNEL_BASE + 0x1)
>
> +#define ELF_KERNEL_MACHINE_TYPE 183
> +
>  inline void irq_disable()
>  {
>  processor::irq_disable();
> diff --git a/arch/x64/arch.hh b/arch/x64/arch.hh
> index 17df5f5c..7bd7517d 100644
> --- a/arch/x64/arch.hh
> +++ b/arch/x64/arch.hh
> @@ -20,6 +20,8 @@ namespace arch {
>  #define INSTR_SIZE_MIN 1
>  #define ELF_IMAGE_START OSV_KERNEL_BASE
>
> +#define ELF_KERNEL_MACHINE_TYPE 62
> +
>  inline void irq_disable()
>  {
>  processor::cli();
> diff --git a/core/elf.cc b/core/elf.cc
> index ffc3187b..ac83e897 100644
> --- a/core/elf.cc
> +++ b/core/elf.cc
> @@ -316,6 +316,9 @@ void file::load_elf_header()
>  throw osv::invalid_elf_error(
>  "bad executable type (only shared-object, PIE or non-PIE
> executable supported)");
>  }
> +if(!(_ehdr.e_machine == ELF_KERNEL_MACHINE_TYPE)) {
>

 _ehdr.e_machine != ELF_KERNEL_MACHINE_TYPE

is probably a simpler way to express the condition.

+throw osv::invalid_elf_error("mismatching machine types");
> +}
>  }
>
>  void file::read(Elf64_Off offset, void* data, size_t size)
> --
> 2.25.1
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/osv-dev/20201129092219.79342-1-samlaberge817%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND9swrsFJF_-5AWSs6jwBJWpDbuM_moegYRTo_S86o2nBw%40mail.gmail.com.

Re: [osv-dev] Looking to Get Started

[Pekka Enberg]

Hi Samuel,

On Tue, Nov 3, 2020 at 2:27 AM Samuel Laberge 
wrote:

> Hi all!
>
> I'm a CS student at the University of Texas at Austin and am really
> interested in the OSv project. For one of my classes I need to contribute
> to an open source project related to virtualization and I've been very
> interested in unikernels.
>
> Are there any tasks/projects that you think could be a good fit for
> getting started contributing to your project? It can even be something
> relatively mundane or tedious that you all may need help on. I'm happy to
> help with anything!
>

You could have a look at issues tagged as "low difficulty" or "medium
difficulty":

https://github.com/cloudius-systems/osv/issues?q=is%3Aissue+is%3Aopen+label%3Alow-difficulty

https://github.com/cloudius-systems/osv/issues?q=is%3Aissue+is%3Aopen+label%3A%22medium+difficulty%22

Waldek, is your roadmap available somewhere? Maybe it should be a markdown
file somewhere in osv.git for newcomers to read?

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND936z8t67EQ45%2BF_6LAv4yc%3DVQQhJhV5niFZ%2BU%3DOu6%2Bog%40mail.gmail.com.

Re: [osv-dev] Problems with socat

[Pekka Enberg]

Hi Rick,

On Thu, Jul 23, 2020 at 4:34 AM Rick Payne  wrote:

>
> Thanks, but the question was more 'why does the OSv socketpair() only
> support SOCK_STREAM, and not SOCK_DGRAM?'. I guess there has to be a
> reason why that decision was made.
>

I doubt there's any technical reason. We likely just did not get to
implementing it.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND_Oba0%2B6ekXuD3UML3rSXP9xc3LXh_M8CgEK8azUL9D9g%40mail.gmail.com.

Re: [osv-dev] Re: raw socket support ?

[Pekka Enberg]

Hi,

On Wed, Jul 15, 2020 at 6:13 PM Waldek Kozaczuk 
wrote:

> Hi,
>
> Unfortunately, I have no idea what it would take to add raw sockets
> support. Please be aware that we maintain another IPV6 branch -
> https://github.com/cloudius-systems/osv/tree/ipv6 - which besides IPV6
> might have better networking support but I doubt it supports raw sockets.
>
> I am also adding Charles Meyers from Spirent who wrote original IPV6
> support to this thread. He may have some thoughts on this matter. Also,
> Spirent has also its own OSv fork - https://github.com/SpirentOrion/osv -
> which has extra stuff/fixes to networking stack (I would like to port some
> of those to the mainline OSv at some point).
>

The TCP/IP stack supports raw sockets (it's the FreeBSD stack after all).

One potential issue is that the Linux socket() compatibility layer is
incorrect. I see that linux_socket() (called by socket() function) has some
support for raw sockets:

https://github.com/cloudius-systems/osv/blob/master/bsd/sys/compat/linux/linux_socket.cc#L619


However, it's bit picky on the "domain" and "protocol" and will ignore what
Frederic attempted to do:

   fd=socket(AF_PACKET,SOCK_RAW,htons(ETH_P_ALL));

I assume if you fix up linux_socket() to do what Linux does, things will
work fine.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-X9cbtnqMZ5STZoysbPLqOr9LFmCtnbMBjLfT8mDQbog%40mail.gmail.com.

Re: [osv-dev] [PATCH] aarch64: until MSI-X support becomes functional enable regular PCI interrupt

[Pekka Enberg]

On Sat, Jul 4, 2020 at 8:18 AM Waldemar Kozaczuk 
wrote:

> The PCI MSI-X support for aarch64 is unfortunately not fully
> functional (please see arch/aarch64/msi.cc for details). This causes
> OSv to hang when running on QEMU in aarch64 emulated mode in both
> virtio-blk and virtio-net waiting to receive PCI interrupt from
> the hypervisor.
>

Acked-by: Pekka Enberg 

Please also open a ticket on osv.git on lack of MSI-X in the AArch64 port.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-X38KBqE6zxTbQpL8Fu0B4oNzpG6YXbVeim3QqCGHd7g%40mail.gmail.com.

Re: [osv-dev] Building customized kernel

[Pekka Enberg]

Hi Dor,

On Thu, Jun 11, 2020 at 8:21 AM Dor Laor  wrote:

> Waldek, can you please describe what's your motivation in shrinking
> the size of the kernel? Compared for example to modern RAM and also
> to the JVM that many times will be there, the current size is already
> 'small enough', isn't it?
>

Not necessarily if the lifetime of a VM is in the order of, say, hundreds
of milliseconds.

My understanding is that these extreme optimizations to reduce size are
motivated by the emergence of light-weight VMs, which allow cloud providers
to launch VMs on-demand for a group of service requests. This allows them
to more efficiently utilize the hardware because an idle VM still consumes
some resources. And in any case, smaller image size anyway means shorter
boot time and more packing of VMs per node.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND_KXX5AoEzGdwrPTnz_7ihxo95AH4uA7i7Nnsptn%3D6vPA%40mail.gmail.com.

Re: [osv-dev] OSv boots and runs simple hello world on Firecracker ARM edition ... on Raspberry PI 4

[Pekka Enberg]

On Fri, May 22, 2020 at 12:26 AM Dor Laor  wrote:

> Kudos Waldek!
>

Indeed, nice work!

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-vurn86y%3DNPu3e98nvnzO6B3aOACLdNpE49Ps2h%3DqCjg%40mail.gmail.com.

Re: [osv-dev] "Iso-UniK: lightweight multi-process unikernel through memory protection keys" based on OSv

[Pekka Enberg]

Hi Waldek,

On Thu, May 14, 2020 at 6:04 PM Waldek Kozaczuk 
wrote:

> https://link.springer.com/article/10.1186/s42400-020-00051-9
>

The same approach of using Intel Memory Protection Keys (MPK) feature was
also suggested in a SPMA'20 workshop paper "The Case for Intra-Unikernel
Isolation" (
https://drive.google.com/file/d/1J-z3nQTKR7hrasAZ69cZ0Mz_ynqjEKEl/view).
Seems like an interesting approach to isolating applications sharing the
same address space...

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND9swxM_QPzyeiLwMPMAe%2BOPHbPmLyDG_H4r-fN-GT%3DBKg%40mail.gmail.com.

Re: [osv-dev] Integrating OpenJDK with OSv to make Java boot and run faster for FaaS

[Pekka Enberg]

On Thu, Mar 19, 2020 at 4:05 PM Pekka Enberg  wrote:
>
> Hi,
>
> On Thu, Mar 19, 2020 at 3:30 PM Yuanqi Li  wrote:
> > Recently we've been looking for new directions. One of the ideas is to 
> > merge JVM and kernel together, so that kernel can expose more system 
> > information (e.g., memory usage and layout, scheduling, network stack) to 
> > JVM. Some potential benefits may include better Java application 
> > performance, better GC performance, faster bootstrapping time, etc.  The 
> > main target will be serverless functions (e.g., AWS Lambda) where most 
> > tasks are short lived function invocations. A slow start of JVM is 
> > prohibitive. An insight is that both the guest OS and JVM are virtual 
> > machines essentially, a lot of language and security checks in JVM may be 
> > redundant given that the guest OS is already well isolated. Memory 
> > allocation and reclaim may also work better since OS can see semantic 
> > information of memory pages. So our first try will be making JVM and kernel 
> > share the same memory management subsystem.
>
> Yeah, I think this is an interesting direction! Merging the JVM and
> the OS was one of the things we were looking to do with OSv, as you
> can see from the JVM memory balloon feature.
>
> > For the new project, I am doing preliminary experiments to collect 
> > performance numbers and estimate potential benefits of the design. OSv is a 
> > very interesting and promising kernel that we'd like to dig more. Its boot 
> > time is blazing fast with Firecracker. I also saw from the original ATC 
> > paper and source code that it has a JVM ballooning mechanism. But I didn't 
> > see how it could work with unmodified JVM. In addition, if I want to 
> > compile both OSv and OpenJDK together from source into a single bootable 
> > image, is there any solutions or resources? I didn't see such documentation 
> > in OSv's github wiki.
>
> I don't remember if we ever built JVM from sources (I'm CC'ing Calle
> for this), but it shouldn't be too hard to do.

(Really CC'ing Calle this time)

>
> The way it works in OSv is that you pass the "image" option to
> "./scripts/build" to build an image with OSv and some applications:
>
> https://github.com/cloudius-systems/osv#building-osv-kernel-and-creating-images
>
> What you need to do is define a OSv "module" or "application" that
> builds JVM from sources. You can find an example of Python being built
> from sources here:
>
> https://github.com/cloudius-systems/osv-apps/tree/master/python27-fromsource
>
> Please note that OSv still consumes the applications as ELF
> executables and shared libraries, and not combine them into one binary
> like other unikernels like Mirage do.
>
> > I know that OSv is designed to support arbitrary unmodifed Linux binaries. 
> > So our idea diverges from OSv's goal. But given the popularity of FaaS and 
> > microservices, a dedicated bootable JVM worths a try. Or we may later find 
> > some low-intrusive disign that doesn't conflict with OSv's goal.
>
> We wanted to run Linux executables out-of-the-box, but did not rule
> out OSv-specific APIs. If you allow yourself to modify the JVM
> sources, then there's really no problem in adding more OSv "system
> calls" (they're really functions) to do what you need to do.
>
> Hope this helps!
>
> - Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND92aBWis5oGx4_AVdM5ACMbdzZPKrghR_BnAJQGBcs%2Bow%40mail.gmail.com.

Re: [osv-dev] Integrating OpenJDK with OSv to make Java boot and run faster for FaaS

[Pekka Enberg]

Hi,

On Thu, Mar 19, 2020 at 3:30 PM Yuanqi Li  wrote:
> Recently we've been looking for new directions. One of the ideas is to merge 
> JVM and kernel together, so that kernel can expose more system information 
> (e.g., memory usage and layout, scheduling, network stack) to JVM. Some 
> potential benefits may include better Java application performance, better GC 
> performance, faster bootstrapping time, etc.  The main target will be 
> serverless functions (e.g., AWS Lambda) where most tasks are short lived 
> function invocations. A slow start of JVM is prohibitive. An insight is that 
> both the guest OS and JVM are virtual machines essentially, a lot of language 
> and security checks in JVM may be redundant given that the guest OS is 
> already well isolated. Memory allocation and reclaim may also work better 
> since OS can see semantic information of memory pages. So our first try will 
> be making JVM and kernel share the same memory management subsystem.

Yeah, I think this is an interesting direction! Merging the JVM and
the OS was one of the things we were looking to do with OSv, as you
can see from the JVM memory balloon feature.

> For the new project, I am doing preliminary experiments to collect 
> performance numbers and estimate potential benefits of the design. OSv is a 
> very interesting and promising kernel that we'd like to dig more. Its boot 
> time is blazing fast with Firecracker. I also saw from the original ATC paper 
> and source code that it has a JVM ballooning mechanism. But I didn't see how 
> it could work with unmodified JVM. In addition, if I want to compile both OSv 
> and OpenJDK together from source into a single bootable image, is there any 
> solutions or resources? I didn't see such documentation in OSv's github wiki.

I don't remember if we ever built JVM from sources (I'm CC'ing Calle
for this), but it shouldn't be too hard to do.

The way it works in OSv is that you pass the "image" option to
"./scripts/build" to build an image with OSv and some applications:

https://github.com/cloudius-systems/osv#building-osv-kernel-and-creating-images

What you need to do is define a OSv "module" or "application" that
builds JVM from sources. You can find an example of Python being built
from sources here:

https://github.com/cloudius-systems/osv-apps/tree/master/python27-fromsource

Please note that OSv still consumes the applications as ELF
executables and shared libraries, and not combine them into one binary
like other unikernels like Mirage do.

> I know that OSv is designed to support arbitrary unmodifed Linux binaries. So 
> our idea diverges from OSv's goal. But given the popularity of FaaS and 
> microservices, a dedicated bootable JVM worths a try. Or we may later find 
> some low-intrusive disign that doesn't conflict with OSv's goal.

We wanted to run Linux executables out-of-the-box, but did not rule
out OSv-specific APIs. If you allow yourself to modify the JVM
sources, then there's really no problem in adding more OSv "system
calls" (they're really functions) to do what you need to do.

Hope this helps!

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND--eYUdV2smCcmD-bHbu7_9ngfdSbSkG7QH5MVymdQOOg%40mail.gmail.com.

Re: [osv-dev] [PATCH] java: move balloon code out of kernel into java-base

[Pekka Enberg]

(Add osv-dev back to CC...)

On Sun, Jan 19, 2020 at 11:16 PM Pekka Enberg  wrote:
>
> On Sun, Jan 19, 2020 at 8:34 AM Waldemar Kozaczuk  
> wrote:
> >
> > The main motivation of this patch is to eliminate compile time
> > dependencies in kernel on JDK headers. It achieves it by moving
> > java balloon shrinker implementation that requires java headers
> > to compile from kernel (java/jvm) to modules/java-base.
> >
> > Given that kernel code in mmu, mempool and mman depended on
> > jvm_balloon_shrinker directly, it was necessary to create new 
> > abstraction/interface -
> > jvm_balloon_api (class with pure virtual methods) - which kernel code
> > would use to interact with jvm balloon code if enabled. The code
> > moved from java/jvm to modules/java-base/balloon would provide
> > implementation of jvm_balloon_api hiding all direct interaction with
> > JNI (Java Native Interface).
> >
> > Signed-off-by: Waldemar Kozaczuk 
>
> Acked-by: Pekka Enberg 

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-rUXFYtAf1LsyvoKsczQ-CkRitMKTcB1ejD%3DhKF5JRYA%40mail.gmail.com.

Re: [osv-dev] [PATCH] Signed-off-by: BassMatt

[Pekka Enberg]

On Thu, Dec 12, 2019 at 10:02 PM BassMatt  wrote:
>
> Main scripts in scripts/ folder updated to use Python3
>
> I went through the scripts detailed in scripts/README and updated them to use 
> Python3. I used the Python "Future" module to provide suggestions, then 
> manually went through and applied the changes. The "Future" module gives 
> suggestions to allow for cross-compatibility between Python2/3, but since it 
> was expressed that only Python3 needed to be supported, I left all that out.
>
> The issue is detailed here:
> https://github.com/cloudius-systems/osv/issues/1056

Looks good, thanks!

Acked-by: Pekka Enberg 

I assume this is you:

https://github.com/BassMatt

Please fix the sign-off to be:

Signed-off-by: Real Name 

as per contributions guide:

https://github.com/cloudius-systems/osv/blob/master/CONTRIBUTING

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND8PC%3DD0ctpO0FxX6mJhBJTxr0d1jiF3-zGRLg_16UCn%3Dw%40mail.gmail.com.

Re: [osv-dev] [PATCH] Fix a file-path typo in the primary README

[Pekka Enberg]

On Tue, Dec 10, 2019 at 6:59 AM Mason Davis  wrote:

> The path is slightly incorrect and can cause issues if someone where to
> copy and paste this code.
>
> Signed-off-by: Mason Davis 
>

Reviewed-by: Pekka Enberg 

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND_%3D7YpdYMNGE%2BPdBT0%2Bco3rabYhkqqnvNfDJBWTdWt4sQ%40mail.gmail.com.

Re: [osv-dev] [PATCH] Manage Dependency with Go Modules and Update APIs to Latest

[Pekka Enberg]

On Mon, Dec 9, 2019 at 3:53 PM kayhaw  wrote:

> As a Go project, Capstan uses godep as its dependency manager,
> however, as of Go 1.11, Go's new dependency management system(Go
> Modules) is introduced officially, more and more Go projects use Go
> Modules to make dependency version information explicit and easier to
> manage, this patch makes Capstan migrate to Go Modules and update
> codes to use the latest APIs, here are details:
>
> - Delete Godeps directory, add go.mod and go.sum file, which are
>   generated by `go mod` command and declare Capstan project is managed
>   with Go Modules now
> - Update packages import path, change urfave/cli from v1 to v2, change
>   cheggaaa/pb from v0 to v3, all packages are newest released version
> - Update code to adapt new APIs according to the old/new API doc
> - Update the installation.md, use Go 1.11+ to build/manage Capstan on
>   Module Mode
>
> See [Go Modules Blogs](https://blog.golang.org/using-go-modules) to
> learn more
>
> Signed-off-by: kayhaw 
>

Waldek, I did not review the details, but the overall approach makes sense:

Acked-by: Pekka Enberg 

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-QAOgw1xB%2BqWjR%3Dej%3DiThj2mGHYShe9bAATFL1P%3DWp4g%40mail.gmail.com.

Re: [osv-dev] Booting Bare Metal

[Pekka Enberg]

On Thu, Dec 5, 2019 at 6:56 AM Waldek Kozaczuk  wrote:

> PS. Do you see anything printed on the console?
>

Nope. I think I should have serial console enabled too to print to host
stdout, but I honestly didn't spend any time debugging this.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-CsXkvP-LqjHoCiii9ZygVM2a39g%3DCugfQZgpjTSu14A%40mail.gmail.com.

Re: [osv-dev] Booting Bare Metal

[Pekka Enberg]

On Tue, Dec 3, 2019 at 5:31 PM Pekka Enberg  wrote:

> Hi Matthew,
>
> On Tue, Dec 3, 2019 at 2:42 PM Matthew Weekley 
> wrote:
>
>> Hello,
>>
>> Sorry, I should’ve been more specific! That is what I am doing. Would it
>> be more help if I gave you the script I am running and my grub.cfg...When I
>> reboot my system and pick that partition in BIOS, then it kernel panics.
>>
>> I’ve been trying to get it to work for some time now but I am stumped.
>> Thanks for your help!
>>
>
> There's not enough information here to help you.
>
> Remember, there could be bugs that prevent booting on bare metals. I would
> suggest that you take the following incremental steps:
>
> (1) Make sure you have the correct process for an ISO image that boots
> Linux with GRUB.
>
> (2) Make sure your ISO image with OSv kernel boots under QEMU/KVM (with
> the same QEMU parameters as run.py uses now).
>
> (3) Make sure your ISO image with OSv boots under QEMU (full emulation
> mode).
>
> (4) Start experimenting with the above ISO image on bare metal.
>
> (5) Insert printfs() to the kernel boot sequence to determine where it
> fails.
>
> (6) Fix any bugs that appear then repeat step (5) until it boots.
>

Btw, with this configuration, I can boot to Linux, but OSv just hangs
(could be lack of VGA or something else):

# GRUB configuration:
mkdir -p build/iso/boot/grub
cat < build/iso/boot/grub/grub.cfg
set timeout=0
set default="0"
menuentry "osv" {
linux /boot/vmlinuz
boot
}
EOF

# Copy kernel image:
cp build/release.x64/vmlinuz.bin build/iso/boot/vmlinuz
#  OR 
cp /boot/vmlinuz-5.2.9-200.fc30.x86_64 build/iso/boot/vmlinuz

# ISO image creation:
grub2-mkrescue -o build/os.iso build/iso/

# Boot under QEMU/KVM:
qemu-system-x86_64 -enable-kvm -cpu host,+x2apic -cdrom build/os.iso

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-0cR7xphzLJQyFgQoo5kpZU7DmO1YfLwDmumYiobSDGQ%40mail.gmail.com.

Re: [osv-dev] Booting Bare Metal

[Pekka Enberg]

Hi Matthew,

On Tue, Dec 3, 2019 at 2:42 PM Matthew Weekley 
wrote:

> Hello,
>
> Sorry, I should’ve been more specific! That is what I am doing. Would it
> be more help if I gave you the script I am running and my grub.cfg...When I
> reboot my system and pick that partition in BIOS, then it kernel panics.
>
> I’ve been trying to get it to work for some time now but I am stumped.
> Thanks for your help!
>

There's not enough information here to help you.

Remember, there could be bugs that prevent booting on bare metals. I would
suggest that you take the following incremental steps:

(1) Make sure you have the correct process for an ISO image that boots
Linux with GRUB.

(2) Make sure your ISO image with OSv kernel boots under QEMU/KVM (with the
same QEMU parameters as run.py uses now).

(3) Make sure your ISO image with OSv boots under QEMU (full emulation
mode).

(4) Start experimenting with the above ISO image on bare metal.

(5) Insert printfs() to the kernel boot sequence to determine where it
fails.

(6) Fix any bugs that appear then repeat step (5) until it boots.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND8pWe%2B_tKUK%3D8Z5uJhz4czgZkhtDco_r2LX7EHGuww9TQ%40mail.gmail.com.

Re: [osv-dev] Booting Bare Metal

[Pekka Enberg]

On Tue, Nov 26, 2019 at 4:50 PM Matthew Weekley 
wrote:

> My current grub is
>
> menuentry “OSv” {
> multiboot /boots/service/release.x64/vmlinuz.bin
>

   ^^^ this needs to "linux", instead of "multiboot".


> boot
> }
>
> This returns “no multiboot header found. You need to load the kernel first”
>

Yes, because there's no Multiboot support in OSv anymore. The "vmlinuz"
file follows the Linux boot protocol described here:

https://www.kernel.org/doc/Documentation/x86/boot.txt

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-DNK5MTx%3D4BTErUqW7q-Nr6OjehQijKhzxCJjHqHXC-g%40mail.gmail.com.

Re: [osv-dev] Booting Bare Metal

[Pekka Enberg]

On Tue, Nov 26, 2019 at 4:14 PM  wrote:

> I've never used vmlinuz to boot, but I see files that pertain to vmlinuz.
> Any suggestions for how to boot bare metal using vmlinuz? Currently looking
> for solutions now. Thanks!
>

Use GRUB, for example. See the previous links to "grub-mkrescue" how to
generate an bootable ISO image. The only difference to Multiboot is that
you need to remember to tell GRUB that the kernel follows the vmlinuz
format, and that's it.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND9wZFAt79M0BgMeoziiQYaKktVG7mdJ-t8YUJKgFrnoqA%40mail.gmail.com.

Re: [osv-dev] Booting Bare Metal

[Pekka Enberg]

On Tue, Nov 26, 2019 at 3:30 PM  wrote:

> I just looked and I do not have any of those files anywhere on my system
> (did the find command).
>

Aah, my tree was not up-to-date. Looks like Waldek removed Multiboot
support:

commit 2a34f9f2acb582b8351c3ca7cac0ac2ad18dfe2a
Author: Waldemar Kozaczuk 
Date:   Wed Jul 24 17:54:56 2019 -0400

Remove obsolete loader.bin build artifact and related source files

At some point long time ago the loader.bin artifact was added
as a way to boot OSv on multiboot bootloaders. Unfortunately it got
abandoned with time,
then resurrected with the commit
e44133273ebdbb305b045b6a74c347a96cab9943 to allow
OSv boot on hyperkit in multiboot mode which never became fully
functional.

Given that now we have Linux-compatible vmlinuz.bin artifact generated
than can be used to boot OSv on hyperkit, there is no need to keep this
arifact
around - the boot process and all kinds of its variation are complex
enough.

Signed-off-by: Waldemar Kozaczuk 

But, GRUB should be able to boot to "vmlinuz.bin" if it's really compatible
with Linux kernel.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-1%2BJ0WjwV_Vv0zVVKET0Eu%3Da57_vYkg%3DuH-%3D0S1Zctvw%40mail.gmail.com.

Re: [osv-dev] Unikernels as highly isolated processes and an idea to implement limited fork/execve in OSv

[Pekka Enberg]

Hi Waldek,

On Tue, Nov 26, 2019 at 12:44 AM Waldek Kozaczuk 
wrote:

> Sometimes it might be more useful to think of unikernels as highly
> isolated processes instead of microVMs with a specialised guest OS. See
> Nabla processes -
> https://acmsocc.github.io/2018/slides/socc18-slides-williams.pdf.
>
> So given that I wonder if it would make sense/be feasible/applicable to
> implement limited fork/execve in OSv. It could work like this:
>
>1. fork() starts new thread T2 - acts as "stub/proxy" to a new remote
>microVM that would be lazily started when execve() is actually called
>2. tid of the new thread T2 would be returned to the caller in the
>thread T1; it would act as PID of the new process
>3. New thread T2 would start with same code as calling thread T1 but
>somehow (???) get 0 as the return value and continue from there (just like
>on normal Posix OS)
>4. T2 typically would call execve() right away
>5.
>   - execve() would somehow communicate with host (how?) to start new
>   child VM with passed in command line, argv
>   - execve() would never return and instead wait unit the child VM
>   terminates
>   - T2 would also somehow listen for some other calls like kill() and
>   pass it to the host that would shutdown child VM
>
> There is obviously a plethora of various IPC mechanisms to communicate
> between processes (shared filesystem, shared memory, named semaphores,
> named semaphores, etc) so how much OSv would need to implement them to make
> such fork/execve useful, I do not know. I am also not sure if that would be
> applicable in any real-world scenarios. Maybe master-worker types of
> workloads (kind of what nginx model is).'
>

You can certainly implement fork()/exec() for an unikernel, and it has been
done before. See, for example, the following paper on Graphene:

  Tsai et al. Cooperation and Security Isolation of Library OSes for
Multi-Process Applications. EuroSys 2014.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.589.1837=rep1=pdf

However, please keep in mind that fork() is used for different kinds of
things, which are not going to be easy to support efficiently using this
kind of model. The way Redis uses fork() for snapshotting comes to mind,
for example.

There is a way to support multi-process environment, though, and that's by
implementing the posix_spawn() system call. I recommend reading the
following paper for a case against fork() and why posix_spawn() makes more
sense in the general case:

  Baumann et al. A fork() in the road. HotOS 2019.
https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND_q5cm093bHuKkPTWPS56ENNh1EcAckH%3DDzGLzUfXamgw%40mail.gmail.com.

Re: [osv-dev] Re: Capstan Build Fail and Fix

[Pekka Enberg]

Hi,

On Mon, Nov 25, 2019 at 6:59 PM Waldek Kozaczuk 
wrote:

> I am not golang expert either. Maybe others, original authors of capstan
> on the list can chime in. I will look into myself. If there is a lot of
> work we might just explicitly stick with v1. Either way, it would be good
> to express the explict dependency on the specific version rather than
> always use master. What is the right way to do in golang?
>

Vendoring (IOW, keeping full copies of the dependencies) used to be the
thing to do, but not sure what Gophers do these days.

But, you probably should anyway just update capstan.git to use the new
APIs, because at some point the older versions will stop working for one
reason or another.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND_S%3DXaG3uJTtYCbyMF10SrYPRwb8JNAS0ifHbPxNqP5mg%40mail.gmail.com.

Re: [osv-dev] Re: [COMMIT osv-apps master] webassembly: add 3 WASM examples - lua, sqlite and nginx - to run on wasmer

[Pekka Enberg]

Hi,

On Mon, Nov 25, 2019 at 6:41 PM Waldek Kozaczuk 
wrote:

> I do not know exactly how wasmer works, but I think on the first run of a
> given wasm file it transforms it into machine code and then stores in the
> cache for the faster start next time. With ZFS images the 2nd time will be
> faster as well. With ROFS the cache is on RAMFS so it transforms the file
> every time.
>

Waldek, your description of Wasmer's ahead-of-time (AOT) compilation with
caching is correct. You can find a high-level overview of their
architecture here:

https://github.com/wasmerio/wasmer/blob/master/docs/architecture.md

and a blog post on the same topic here:

https://medium.com/wasmer/a-webassembly-compiler-tale-9ef37aa3b537

Nice work getting Wasmer to run on OSv!

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-vRnvaA3_ztPFW2r7hWts6YEMfZxZ6UBdYPs1a5To3nQ%40mail.gmail.com.

Re: [osv-dev] CI Implementation/Contribution

[Pekka Enberg]

Hi Matthew,

On Wed, Nov 20, 2019 at 7:29 AM Matthew Pabst 
wrote:

> A couple of peers and I from the University of Texas at Austin are
> interested in contributing to OSv, and we thought a good place to start
> would be implementing some basic CI infrastructure (
> https://github.com/cloudius-systems/osv/issues/1044). A couple of us have
> experience with CI/DevOps activities, so it shouldn't be too difficult.
>
> We have a few concerns though:
>
>- The Dockerfile is missing make, so we'd have to add that to run the
>build scripts
>
>
So why not send a patch or a pull request to add "make" as a dependency to
the Dockerfile?


>
>- When we try to run scripts/build check  with the latest commit
>(7d4035f8c81e263340df1ba4c5db305714a1), tracing_smoke_test fails
>on the tcpdump section
>
>
I will defer this question to Waldek and Nadav.

>
>- The testing script fails immediately if any test fails, rather than
>collecting the output of all tests
>
>
I think that was by design because the assumption is that if one of the
tests fails, you are likely to see a ripple effect of other tests failing
too. But I don't see any reason why you couldn't add a command-line switch
to disable fail-fast.


> After this issue, we plan to take a look at a handful of these beefier
> issues - not sure which one(s) yet, but some guidance would be appreciated:
>
>- https://github.com/cloudius-systems/osv/issues/1056
>
>
This should be pretty straight-forward, no? Just go through all the Python
scripts, switch interpreter to "python3", and address any compatibility
issues.


>
>- https://github.com/cloudius-systems/osv/issues/1008
>
>
You have two options to go about this: (1) implement a 9p filesystem from
scratch or (2) port a 9p filesystem from FreeBSD, for example (assuming one
exists).

For (1), you can get started by looking at the ramfs filesystem:
https://github.com/cloudius-systems/osv/tree/master/fs/ramfs

For (2), have a look at what we did with the ZFS filesystem. It's also
ported from FreeBSD and we added a bunch of wrapper code to make that
happen. So it could be that a lot of infrastructure already exists for the
9p filesystem.


>
>- https://github.com/cloudius-systems/osv/issues/1013
>
>
Waldek and Nadav will probably have a better answer for this.

Regards,

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND_EO%3D3tjucGKw-dPqswJ%2Bs8fNLO-_Va7SOTuKU7016cnw%40mail.gmail.com.

Re: [osv-dev] Projects for Students at the University of Texas in Austin

[Pekka Enberg]

Hi,

On Thu, Oct 24, 2019 at 11:41 PM Jie Hao Liao  wrote:

> For whom may this concern,
>
> I'm Jay, one of the students at the University of Texas in Austin majoring
> in Computer Science. This fall 2019 semester I'm taking a Virtualization
>  course for understanding and
> building Hypervisors, Containers, and Unikernels. We would like to work on
> a project related to building features for Unikernels for this class.
>
> We're a group of 3 people that'll be contributing to the project. We have
> learned about some different implementations of Unikernels, but we did not
> have any implementation experience for Unikernels. We have built core
> components of a working educational JOS hypervisor in this class. The
> deadline for finishing the project is December 12th. Could anyone please
> recommend us a list of possible Unikernel features projects for OSv or some
> point-of-contact for finding some good starter projects on OSv that can
> last 2 months?
>

I am sure Waldek and Nadav will have even better ideas, but I found these
two issues on GitHub that might be interesting. Both of them are related to
porting OSv to run on new hypervisors:

https://github.com/cloudius-systems/osv/issues/948

https://github.com/cloudius-systems/osv/issues/924

Another interesting project might be porting OSv to RISC-V. Linux recently
added (some) support for RISC-V in KVM so you should be able to get
something up and running.

Hope this helps!

Regards,

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/osv-dev/CAGUyND-6Gi-5gGSVHqRrw0PeCqzyxtsmma42czEwo%3D1OCUSHUw%40mail.gmail.com.

Re: [osv-dev] How to compose a vbox platform image with new capstan

[Pekka Enberg]

Hi,

I don't think Capstan supports building VirtualBox images, but you can use
the "qemu-img convert -O vdi" command to convert a QCOW2 (QEMU's format)
image to a VirtualBox image.

- Pekka

On Tue, Apr 16, 2019 at 9:03 AM Roberto Battistoni 
wrote:

> Hi guys!
>
> Any advice? Please I am very close to conclude my experimentations but
> QEMU is too slow and I need to do other tests with more efficient platform.
>
> thanks so much
>
> r
>
> Il giorno 14 apr 2019, alle ore 08:13, robertob 
> ha scritto:
>
> I made package.yaml that correctly run with default QEMU platform, but now
> I do not understand how to build image for virtualbox platform. When I
> build the image (capstan package compose uni) I see every time that the
> QEMU image was created.
>
> I tried this package.yaml file with "platform: vbox" but noting
>
> name: uni
> title: Uni
> platform: vbox
>
> require:
> - openjdk8-zulu-compact3-with-java-beans
> - osv.httpserver-html5-gui-and-cli
> - osv.run-java
> - osv.cli
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "OSv Development" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/osv-dev/-fwSG-MY9Ks/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> osv-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] Capstan

[Pekka Enberg]

Hi,

On Wed, Apr 3, 2019 at 4:24 PM Waldek Kozaczuk  wrote:

> In near future I am planning to add a capability to pull artifacts from
> github repo. I am planning to create pull request for that.
>
> Can I count on someone to review my pull requests? Any volunteers?
>

No promises, but you can ping me for reviews.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] OSv vs Docker vs Linux networking performance comparison

[Pekka Enberg]

Hi Waldek,

On Thu, Mar 28, 2019 at 12:49 AM Waldek Kozaczuk 
wrote:

> Some questions about the evaluation setup and measurements:
>>>
>>> - Did you establish a baseline with bare metal configuration?
>>>
>> How would I create baseline with with bare metal configuration for 1, 2,
> 4 CPUs? With docker or qemu I can specify number of cpus.
>

You can use the "taskset" command to restrict process to run on specific
CPUs. But I think a 4 CPU bare metal baseline is sufficient because then
you know what is the maximum expected throughput.


>
> - Did you measure CPU utilization during the throughput tests? This is
>>> important because you could be hitting CPU limits with QEMU and Firecracker
>>> because of software processing needed by virtualized networking.
>>>
>> Nothing rigorous. I has mpstat running and I could see that during 1 and
> 2 cpu tests they were pretty highly utilized (80-90%) but only 40-50% for 4
> cpu tests. But nothing I recorded.
>

I would encourage you to run something like "vmstat" or "sar" in the
background to obtain average CPU utilization for the run to be able to
compare the results.

The drop in CPU utilization suggests that you're bound by the network. How
 big are the HTTP requests and responses your test is generating? You could
be hitting the ~110 MB/s bandwidth limit of a 1 GbE NIC. Also, note that
40-50% CPU utilization is quite low for a throughput test so you're mostly
seeing the impact of latency here. This is where network bridge
configuration becomes relevant too. If you have more layers the higher the
latency is going to be.


>
>> - Are the QEMU and Firecracker tests using virtio or vhost?
>>>
>> I thought OSv only support virtio. Sorry to be ignorant. I heard the
> terms but what is actually the difference between vhost and virtio?
>

Sorry for not being explicit. Virtio is the guest/hypervisor I/O interface,
which OSv also supports. However, there are two implementations of the I/O
model: virtio (in host userspace) and vhost (in host kernel). You can think
of vhost as a host kernel accelerator for virtio:

http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html

The main difference is that vhost is supposed to be faster than virtio
because it reduces VM exits.


>
>
>
>> - Is Docker also configured to use the bridge device? If not, QEMU and
>>> Firecracker also have some additional overheads from the bridging.
>>>
>> I need to check. Per this -
> https://raw.githubusercontent.com/wkozaczuk/unikernels-v-containers/master/run-rest-in-docker.shI
> am sure - I would expose container port to the host. So I think I was
> bypassing the bridge.
>
> BTW is there a way to run OSv on QEMU without a bridge to make it visible
> on LAN?
>

AFAICT, this would require either device assignment or SR-IOV, but neither
is supported by OSv due to lack of (real) hardware device drivers.


>
> - Is multiqueue enabled for QEMU and Firecracker? If not, this would
>>> limit the ability to leverage multiple vCPUs.
>>>
>> No idea what you are talking about ;-)
>

IIRC, there's a "queues" option you pass to "-netdev" with QEMU. No idea
about Firecracker.

That said, we have the following comment in virtio-net drivers:

//
// We currently have only a single TX queue. Select a proper TXq here
when
// we implement a multi-queue.
//

So perhaps we don't even support multiqueue in OSv at the moment...

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] Capstan

[Pekka Enberg]

Hello,

On Thu, Mar 28, 2019 at 3:20 AM Waldek Kozaczuk 
wrote:

> All,
>
> Please let me know if you think it is NOT a good idea to merge back
> capstan repo back to cloudius-systems/capstan. Otherwise I propose we do so.
>
> I have never done such a merge on github. If anybody has any suggestions
> what the best procedure it should be let me know. I would obviously need a
> permission to do os. Create a merge request?
>
> Also I hope there are no licencing issues with merging up all the content
> up. Should we retain some Mikelangelo project references?
>

I would just do a "git pull" since there are no conflicts:

[penberg@nero ~]$ git clone g...@github.com:cloudius-systems/capstan.git
Cloning into 'capstan'...
remote: Enumerating objects: 1774, done.
remote: Total 1774 (delta 0), reused 0 (delta 0), pack-reused 1774
Receiving objects: 100% (1774/1774), 329.55 KiB | 1.28 MiB/s, done.
Resolving deltas: 100% (1019/1019), done.

[penberg@nero ~]$ cd capstan/
[penberg@nero capstan]$ git pull
https://github.com/mikelangelo-project/capstan.git master
remote: Enumerating objects: 143, done.
remote: Counting objects: 100% (143/143), done.
remote: Total 1401 (delta 142), reused 142 (delta 142), pack-reused 1258
Receiving objects: 100% (1401/1401), 509.60 KiB | 1.38 MiB/s, done.
Resolving deltas: 100% (873/873), completed with 32 local objects.
>From https://github.com/mikelangelo-project/capstan
 * branchmaster -> FETCH_HEAD
Updating 47fff91..5af44b4

The "git push", of course. Anything that needs to be addressed can be fixed
incrementally.

There are no licensing concerns because all the modifications are
contributed under the same license. Please *do* keep the Mikelangelo
references since they are part of the development history.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] OSv vs Docker vs Linux networking performance comparison

[Pekka Enberg]

On Wed, Mar 27, 2019 at 10:48 AM Pekka Enberg  wrote:

> Hi Waldek!
>
> On Wed, Mar 27, 2019 at 12:29 AM Waldek Kozaczuk 
> wrote:
>
>> Last week I spent some time investigating OSv performance and comparing
>> it to Docker and Linux guests.
>>
>
> Nice!
>
> On Wed, Mar 27, 2019 at 12:29 AM Waldek Kozaczuk 
> wrote:
>
>> The test setup looked like this:
>>
>> *Host:*
>>
>>- MacBook Pro with Intel i7 4 cores CPU with hyperthreading (8 cpus
>>reported by lscpu) with 16GB of RAM with Ubuntu 18.10 on it
>>- firecracker 0.15.0
>>- QEMU 2.12.0
>>
>>
>> *Client machine:*
>>
>>- similar to the one above with wrk as a test client firing requests
>>using 10 threads and 100 open connections for 30 seconds in 3 series one 
>> by
>>one (please see this test script -
>>
>> https://github.com/wkozaczuk/unikernels-v-containers/blob/master/test-restapi-with-wrk.sh
>>).
>>- wrk by default uses Keep-Alive for http connections so TCP
>>handshake is minimal
>>
>> The host and client machine were connected directly to 1 GBit ethernet
>> switch and host exposed guest IP using a bridged TAP nic (please see the
>> script used -
>> https://raw.githubusercontent.com/cloudius-systems/osv/master/scripts/setup-external-bridge.sh
>> ).
>>
>> You can find scripts to start applications on OSv and docker here -
>> https://github.com/wkozaczuk/unikernels-v-containers (run* scripts).
>> Please note --cpu-set parameter used in docker script to limit number of
>> CPUs.
>>
>> You can find detailed results under
>> https://github.com/wkozaczuk/unikernels-v-containers/tree/master/test_results/remote
>> .
>>
>
> Some questions about the evaluation setup and measurements:
>
> - Did you establish a baseline with bare metal configuration?
> - Did you measure CPU utilization during the throughput tests? This is
> important because you could be hitting CPU limits with QEMU and Firecracker
> because of software processing needed by virtualized networking.
> - Are the QEMU and Firecracker tests using virtio or vhost?
> - Is Docker also configured to use the bridge device? If not, QEMU and
> Firecracker also have some additional overheads from the bridging.
> - Is multiqueue enabled for QEMU and Firecracker? If not, this would
> limit the ability to leverage multiple vCPUs.
> - Is QEMU or Firecracker setting CPU affinity for the vCPU threads? If
> not, two or more vCPUs could be running on the same physical CPU, which
> obviously limits throughput.
>

Oh forgot the obvious:

- Is CPU scaling governor set to performance? Also, if the CPU has
TurboBoost, is it disabled?

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] OSv vs Docker vs Linux networking performance comparison

[Pekka Enberg]

Hi Waldek!

On Wed, Mar 27, 2019 at 12:29 AM Waldek Kozaczuk 
wrote:

> Last week I spent some time investigating OSv performance and comparing it
> to Docker and Linux guests.
>

Nice!

On Wed, Mar 27, 2019 at 12:29 AM Waldek Kozaczuk 
wrote:

> The test setup looked like this:
>
> *Host:*
>
>- MacBook Pro with Intel i7 4 cores CPU with hyperthreading (8 cpus
>reported by lscpu) with 16GB of RAM with Ubuntu 18.10 on it
>- firecracker 0.15.0
>- QEMU 2.12.0
>
>
> *Client machine:*
>
>- similar to the one above with wrk as a test client firing requests
>using 10 threads and 100 open connections for 30 seconds in 3 series one by
>one (please see this test script -
>
> https://github.com/wkozaczuk/unikernels-v-containers/blob/master/test-restapi-with-wrk.sh
>).
>- wrk by default uses Keep-Alive for http connections so TCP handshake
>is minimal
>
> The host and client machine were connected directly to 1 GBit ethernet
> switch and host exposed guest IP using a bridged TAP nic (please see the
> script used -
> https://raw.githubusercontent.com/cloudius-systems/osv/master/scripts/setup-external-bridge.sh
> ).
>
> You can find scripts to start applications on OSv and docker here -
> https://github.com/wkozaczuk/unikernels-v-containers (run* scripts).
> Please note --cpu-set parameter used in docker script to limit number of
> CPUs.
>
> You can find detailed results under
> https://github.com/wkozaczuk/unikernels-v-containers/tree/master/test_results/remote
> .
>

Some questions about the evaluation setup and measurements:

- Did you establish a baseline with bare metal configuration?
- Did you measure CPU utilization during the throughput tests? This is
important because you could be hitting CPU limits with QEMU and Firecracker
because of software processing needed by virtualized networking.
- Are the QEMU and Firecracker tests using virtio or vhost?
- Is Docker also configured to use the bridge device? If not, QEMU and
Firecracker also have some additional overheads from the bridging.
- Is multiqueue enabled for QEMU and Firecracker? If not, this would limit
the ability to leverage multiple vCPUs.
- Is QEMU or Firecracker setting CPU affinity for the vCPU threads? If not,
two or more vCPUs could be running on the same physical CPU, which
obviously limits throughput.

Regards,

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] Updating http://osv.io/

[Pekka Enberg]

On Mon, Mar 25, 2019 at 10:37 AM Nadav Har'El  wrote:

>
> On Mon, Mar 25, 2019 at 8:38 AM Tzach Livyatan  wrote:
>
>>
>> On Mon, Mar 25, 2019 at 8:17 AM Dor Laor  wrote:
>>
>>> Waldek, I added you to the blog as admin, a Firecracked blog will be
>>> awesome.
>>> I failed to locate the main site itself, Tzach, Pekka, do you remember?
>>>
>>
>> This is the repo for the site
>> https://github.com/cloudius-systems/osv
>>
>
> This is, of course, not the site, but the code
>

The repository for the OSv home page is here:

https://github.com/osv-io/osv-io.github.io

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] Re: Unikernel performance review paper inquiry

[Pekka Enberg]

Hi,

On Thu, Mar 14, 2019 at 5:11 AM Waldek Kozaczuk 
wrote:

> I wonder if anyone has had chance to read this paper. I would like to see
> what others think about reasons OSv thread scheduler does not scale well
> with number of vCPUs.
>

I am not sure that the scheduler is to blame here.

It is possible (even likely) that applications are impacted by different
problems. For example, the paper does not mention if the Python Flask-based
application is configured to use multi-threading mode or not. As the
performance metrics don't change between single core and multicore, it
suggests that perhaps Flask was run in the default single-threaded mode.
The paper mentions that they allow Go to manage the worker thread pool size
itself, but it's not clear what APIs Go uses to discover the number of CPUs
and whether OSv implements this correctly, which could lead to Go using a
thread pool that is too small or large.

The Java application numbers are most interesting because most of the
performance work for OSv was done with native and JVM applications (for
example, Apache Cassandra) and the combination is expected to work.
Unsurprisingly, for Java application, you see some improvement with
multicore, although OSv does not perform as well as Docker. However, the
most interesting part of the evaluation results is the following
observation:

“It should be noted that in all cases the combined CPU load of all cores
was between 40 and 50% during testing, indicating that a lot of cycles were
being wasted on simply getting threads to run at all, and not enough on
actually running them.”

The first thing I would look for here is if there's a bottleneck in OSv's
networking. In particular, if Xen network driver and device multi-queue are
enabled and packet steering (packet distribution to different CPUs) is
working correctly. Furthermore, I would look for any locking in the
networking stack that serializes packet receive or transmission. Also, one
area that JVM-based applications are often affected is locking. So it might
make sense to look at OSv's pthread mutex implementation, for example.

It would be interesting for someone to attempt to first reproduce the
results and then use OSv's tracing to gain more understanding of what is
actually going on:

https://github.com/cloudius-systems/osv/wiki/Trace-analysis-using-trace.py

Regards,

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] [COMMIT osv master] elf: support PT_GNU_PROPERTY segment type

[Pekka Enberg]

On Fri, Mar 8, 2019 at 10:42 AM Commit Bot  wrote:

> From: Nadav Har'El 
> Committer: Nadav Har'El 
> Branch: master
>
> elf: support PT_GNU_PROPERTY segment type
>
> Recently, binutils added yet another segment type, called
> "PT_GNU_PROPERTY":
>
> https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=0a59decbb81676ac30deede1bb6b6e241cd75502
>
> Since OSv's linker currently aborts when it sees an unknown segment type,
> we saw crashes when loading some specific shared objects, like a recent
> Java build from Fedora.
>
> This patch adds the missing segment type (and ignores it), and also
> improves
> the error message in case in the future we encounter more newly invented
> segment types.
>
> Fixes #1030.
>
> Signed-off-by: Nadav Har'El 
>
> ---
> diff --git a/core/elf.cc b/core/elf.cc
> --- a/core/elf.cc
> +++ b/core/elf.cc
> @@ -423,15 +423,15 @@ void object::load_segments()
>   case PT_GNU_RELRO:
>   case PT_GNU_EH_FRAME:
>   case PT_PAX_FLAGS:
> +case PT_GNU_PROPERTY:
>   break;
>   case PT_TLS:
>   _tls_segment = _base + phdr.p_vaddr;
>   _tls_init_size = phdr.p_filesz;
>   _tls_uninit_size = phdr.p_memsz - phdr.p_filesz;
>   break;
>   default:
> -abort();
> -throw osv::invalid_elf_error("bad p_type");
> +abort("Unknown p_type in executable %s: %d\n", pathname(),
> phdr.p_type);
>

Can't we always ignore unknown segment types, but print the above error
message as a warning?

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] OSv fully and officially supported on Firecracker !!!

[Pekka Enberg]

On Wed, Mar 6, 2019 at 6:24 AM Waldek Kozaczuk  wrote:

> With couple of last patches applied yesterday we finally made OSv properly
> and fully run on Firecracker. This includes ability to run arbitrary images
> (zfs, rofs or ramfs) with networking and SMP supported as well.
>

Nice work, Waldek!

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] Does OSv have its own scheduler?

[Pekka Enberg]

Hello,

On Sun, Feb 17, 2019 at 11:45 PM Joe Duarte  wrote:

> Hi all -- Does OSv have its own scheduler, or does it rely on the Xen
> scheduler?
>
> I guess this might be similar to asking if OSv has threads. I have no
> idea. I'm not sure what being a unikernel is supposed to imply for threads.
>

OSv indeed has as a scheduler to support threads. You can find a more
detailed discussion in the OSv paper, for example:

https://www.usenix.org/system/files/conference/atc14/atc14-paper-kivity.pdf

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [osv-dev] Re: [RFC] Remaining work to support OSv on firecracker

[Pekka Enberg]

On Mon, Feb 11, 2019 at 10:54 AM Nadav Har'El  wrote:

> Maybe you can even add to run.py a firecracker hypersor options, so if
> someone just installs this hypervisor he can use run.py?
>

That's a very good idea!

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] Refactor virtio layer to support modern PCI and mmio devices

[Pekka Enberg]

On Wed, Jan 16, 2019 at 6:02 PM Nadav Har'El  wrote:

>
> On Wed, Jan 16, 2019 at 5:53 PM Waldek Kozaczuk 
> wrote:
>
>> On Wednesday, January 16, 2019 at 1:29:13 AM UTC-5, Pekka Enberg wrote:
>>>
>>> But for the overall approach:
>>>
>> ??? I thought you were going to say something more here :-)
>>
>>>
>>> Acked-by: Pekka Enberg 
>>>
>>
> I think Pekka meant he "Acked" the "overall approach" :-)
>

Exactly. Assuming it works, I think the patch should go in. However, I did
not review it in detail, hence "Acked-by" and not "Reviewed-by".

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] Refactor virtio layer to support modern PCI and mmio devices

[Pekka Enberg]

Hello,

On Wed, Jan 16, 2019 at 7:21 AM Waldemar Kozaczuk 
wrote:

> This patch refactors virtio layer in order to support
> modern PCI and mmio devices. It is essentially a step 0
> in the list of tasks to make OSv boot on AWS firecracker.
>
> This patch introduces virtio_device class that abstracts
> virtio transport layer. That way virtio_driver class and
> its specializations can interact with host without being
> tied to a specific transport (PCI, mmio, etc).
>
> The virtio_pci_device, virtio_legacy_pci_device and
> virtio_modern_pci_device
> classes define specializations of virtio_device for PCI transport.
> This patch effectively moves all PCI-related logic from virtio_driver
> to virtio_legacy_pci_device classi and effectively virtio drivers
> transport agnostic. Followup patches will provide implementations
> of virtio_modern_pci_device and future virtio_mmio_device.
>
> The interrupt_factory struct defines a mechanism used
> by virtio drivers to specify particulars of creating/registering
> proper interrupt logic for each transport.
>

You coud make this patch even simpler to review by deferring the
introduction of virtio_modern_pci_device in a follow up patch.

But for the overall approach:

Acked-by: Pekka Enberg 

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [RFC] Support firecracker

[Pekka Enberg]

On Sat, Jan 5, 2019 at 9:58 AM Pekka Enberg  wrote:

>
>
> On Sat, Jan 5, 2019 at 9:43 AM Waldek Kozaczuk 
> wrote:
>
>> Indeed I have come across some incompatibility issues. Mostly around
>> configuration but nothing very fundamental. You can see in some of my
>> comments.
>>
>
> Right, I missed the comments because the diff is somewhat large. ;-)
>
> I would suggest to do Firecracker support in the following (largely
> independent) steps:
>
> - Add support for modern virtio interfaces. Linux has a split driver base
> model here, which we could also follow. I.e., have separate VirtioModern
> and VirtioLegacy base classes that encapsulate the differences.
>

Btw, this can be also tested with QEMU by passing the "disable-modern=off"
and "disable-legacy=on" options to the "-device" option that defines the
virtio devices.

- Pekka

>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [RFC] Support firecracker

[Pekka Enberg]

On Sat, Jan 5, 2019 at 9:43 AM Waldek Kozaczuk  wrote:

> Indeed I have come across some incompatibility issues. Mostly around
> configuration but nothing very fundamental. You can see in some of my
> comments.
>

Right, I missed the comments because the diff is somewhat large. ;-)

I would suggest to do Firecracker support in the following (largely
independent) steps:

- Add support for modern virtio interfaces. Linux has a split driver base
model here, which we could also follow. I.e., have separate VirtioModern
and VirtioLegacy base classes that encapsulate the differences.

- Add support for VirtIO MMIO. Most of this is restructuring the drivers to
use transport-independent interfaces and then have VirtioPCI and VirtioMMIO
classes that implement transport-specific logic.

- Add Intel mptable (used by Firecracker) support as an alternative to ACPI
hardware discovery.

- Add any misc Firecraker related things that are still needed to make OSv
boot and run.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: OSv boots on AWS firecracker in 7ms (seven)

[Pekka Enberg]

On Sat, Jan 5, 2019 at 9:04 AM Waldek Kozaczuk  wrote:

> And one more thing I forgot to mention. I do not know much about ARM
> architecture but I think the patch I have just sent may help us provide
> virtio implementation for ARM as well, no?
>

I think ARM64 also supports virtio-pci these days:

https://groups.google.com/a/groups.riscv.org/forum/#!topic/sw-dev/3MLIq6rxa9Q

https://wiki.qemu.org/Documentation/Platforms/ARM#Generic_ARM_system_emulation_with_the_virt_machine

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [RFC] Support firecracker

[Pekka Enberg]

Hi Waldek,

On Sat, Jan 5, 2019 at 8:47 AM Waldek Kozaczuk  wrote:

> Please bear in mind it is a RFC polluted with hacks and many TODO comments
> that raise many questions.
>
> Feel free to review it especially if you have ideas on how to better
> restructure the code (I left some comments about it) and if you see any
> obvious bugs.
>

Looks like the bulk of the changes are related to supporting VirtIO devices
via MMIO because Firecracker does not support PCI?

Btw, I think that Firecracker implements the "modern" VirtioIO interface
(i.e., version 1.0 and later) but I think OSv's VirtIO drivers are still
using the "legacy" interfaces. Did you hit any compatibility issues because
of this?

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to test the schedule latency?

[Pekka Enberg]

On Thu, Oct 11, 2018 at 12:22 PM Avi Kivity  wrote:

>
> On 11/10/2018 12.14, Nadav Har'El wrote:
>
> On Thu, Oct 11, 2018 at 12:10 PM Pekka Enberg 
> wrote:
>
>>
>>
>> On Thu, Oct 11, 2018 at 11:50 AM Nadav Har'El  wrote:
>>
>>>
>>> tests/misc-ctxsw.cc
>>>
>>> To run it in OSv you can do "scripts/build image=tests" and then
>>> "scripts/run.py -e tests/misc-ctxsw.so"
>>>
>>> What this test does is to create two threads (on the same core, or on
>>> two different cores) which alternate waking the other thread and then
>>> waiting (via a condition variable).
>>> Strangely I just tested this again on my laptop (which is newer than the
>>> paper), and got (for the colocated case) in OSv is around 400ns, and in
>>> Linux around 1800ns, so the difference is even more pronounced today.
>>>
>>
>> Perhaps you are seeing the effects of Meltdown mitigation in Linux
>> context switching paths?
>>
>
> Perhaps, I didn't look into this carefully, I just ran it now to make sure
> I was pointing the OP to the right test :-)
> Does meltdown mitigation have a performance cost even when switching
> between two threads belonging to the same process?
>
>
>
> Yes, the kernel and userspace run in different address spaces (CR3 switch
> on every kernel entry and exit). It's not a thread context switch cost, but
> a syscall cost.
>
Right, but AFAICT, it's also an interrupt cost. So whatever the mechanism
that causes the context switch to another thread, you need to switch to
kernel CR3 and to the userspace thread CR3.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to test the schedule latency?

[Pekka Enberg]

On Thu, Oct 11, 2018 at 12:14 PM Nadav Har'El  wrote:

> On Thu, Oct 11, 2018 at 12:10 PM Pekka Enberg 
> wrote:
>
>>
>>
>> On Thu, Oct 11, 2018 at 11:50 AM Nadav Har'El  wrote:
>>
>>>
>>> tests/misc-ctxsw.cc
>>>
>>> To run it in OSv you can do "scripts/build image=tests" and then
>>> "scripts/run.py -e tests/misc-ctxsw.so"
>>>
>>> What this test does is to create two threads (on the same core, or on
>>> two different cores) which alternate waking the other thread and then
>>> waiting (via a condition variable).
>>> Strangely I just tested this again on my laptop (which is newer than the
>>> paper), and got (for the colocated case) in OSv is around 400ns, and in
>>> Linux around 1800ns, so the difference is even more pronounced today.
>>>
>>
>> Perhaps you are seeing the effects of Meltdown mitigation in Linux
>> context switching paths?
>>
>
> Perhaps, I didn't look into this carefully, I just ran it now to make sure
> I was pointing the OP to the right test :-)
> Does meltdown mitigation have a performance cost even when switching
> between two threads belonging to the same process?
>

Yes, that's my understanding.

Linux mitigates against Meltdown with kernel page table isolation.
Userspace processes no longer have kernel memory mapped in the upper half
of the address space (to avoid side-channel attacks from reading arbitrary
mapped memory), which means that the kernel needs to switch page tables to
access kernel memory. Search for SWITCH_TO_KERNEL_CR3 for system call entry
in arch/x86/entry/entry_64.S, for example.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to test the schedule latency?

[Pekka Enberg]

On Thu, Oct 11, 2018 at 11:50 AM Nadav Har'El  wrote:

>
> On Thu, Oct 11, 2018 at 10:45 AM Qixuan Wu  wrote:
>
>> Hi All,
>>
>> I saw the paper,  it said schedule latency is less than linux thread,
>> about 328ns.
>>
>> Guest OS
>>
>> Colocated
>>
>> Apart
>>
>> Linux
>>
>> 905 ns
>>
>> 13148 ns
>>
>> OSv
>>
>> 328 ns
>>
>> 1402 ns
>>
>> Anyone can tell where is the test case code ?
>>
>
> tests/misc-ctxsw.cc
>
> To run it in OSv you can do "scripts/build image=tests" and then
> "scripts/run.py -e tests/misc-ctxsw.so"
>
> What this test does is to create two threads (on the same core, or on two
> different cores) which alternate waking the other thread and then waiting
> (via a condition variable).
> Strangely I just tested this again on my laptop (which is newer than the
> paper), and got (for the colocated case) in OSv is around 400ns, and in
> Linux around 1800ns, so the difference is even more pronounced today.
>

Perhaps you are seeing the effects of Meltdown mitigation in Linux context
switching paths?

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: gvisor - google user-space kernel in go

[Pekka Enberg]

Hi Nadav,

On Thu, Sep 20, 2018 at 11:03 AM, Nadav Har'El  wrote:

> I think Qixuan may be referring to the fact that OSv today can only run
> position-independent executables (shared objects and PIE), and nor
> "regular" position-dependent executables,
> which causes us in many applications in apps/ to modify the Makefile to
> add "-shared", "-pie", or something like that, requiring recompilation.
>

Right, makes sense. But I think the point still stands, there's no actual
requirement of *compilation*, just making sure the executable is
position-independent.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: gvisor - google user-space kernel in go

[Pekka Enberg]

On Thu, Sep 20, 2018 at 9:29 AM, Qixuan Wu  wrote:

>
>
> 在 2018年9月19日星期三 UTC+8下午9:25:15，Pekka Enberg写道：
>>
>>
>>
>> On Wed, Sep 19, 2018 at 4:18 AM, Qixuan Wu  wrote:
>>
>>>
>>>- claim to run unmodified Linux executable like OSv does
>>>
>>> gVisor can run the C application without recompiling, but application
>>> only can run on OSv with recompiling.
>>>
>>
>> OSv is quite capable of running unmodified Linux executables, as long as
>> they use the subset of Linux ABI OSv actually supports (IOW, no fork(), for
>> example).
>>
>> But application need to be recompiled on OSv, it's better to support to
> be without recompiling.
>

No, OSv does _not_ require applications to be recompiled. What makes you
think it does?

OSv implements a subset of the Linux system call ABI and has a built-in ELF
loader and linker. This allows OSv to run *unmodified* Linux binaries. Of
course, if your application uses parts of the Linux ABI that are not
supported, such as fork, they will not run very long. But OSv is capable of
running unmodified Linux binaries for applications that restrict themselves
to the subset of the ABI that OSv supports.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: gvisor - google user-space kernel in go

[Pekka Enberg]

On Wed, Sep 19, 2018 at 4:18 AM, Qixuan Wu  wrote:

>
>- claim to run unmodified Linux executable like OSv does
>
> gVisor can run the C application without recompiling, but application only
> can run on OSv with recompiling.
>

OSv is quite capable of running unmodified Linux executables, as long as
they use the subset of Linux ABI OSv actually supports (IOW, no fork(), for
example).

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Running OSv on kvmtool

[Pekka Enberg]

Hi,

On Wed, Apr 11, 2018 at 6:08 PM, Waldek Kozaczuk 
wrote:

> Do you know if kvmtool supports multiboot? If so which version?
>

It does not support Multiboot at all. The main problem with booting OSv is
indeed the loading part. kvmtool on x86 only supports Linux bzImage. The
MIPS port supports ELF so perhaps that could be included for x86 to to boot
OSv.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [COMMIT osv master] update apps submodule

[Pekka Enberg]

On Fri, Jul 7, 2017 at 1:08 AM, Commit Bot  wrote:
> From: Nadav Har'El 
> Committer: Nadav Har'El 
> Branch: master
>
> update apps submodule
>
> Signed-off-by: Nadav Har'El 

Nadav, it's nice to have "git submodule summary" in the commit message
for submodule updates so that you can quickly see what was changed.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] loader: do not kill httpserver prematurely

[Pekka Enberg]

On Mon, Dec 26, 2016 at 12:04 AM, Nadav Har'El  wrote:
> Commit e270cf9c (more than two years ago) was well-intentioned, but
> introduced a new bug: we no longer wait for the background programs
> to finish before stopping httpserver. So for example if the command
> line is something like
>
>/cli/cli.so & finish_quickly.so
>
> And finish_quickly.so finishes quickly, we do not wait for cli.so to
> finish before killing httpserver.
>
> One user discovered that because of that, the image created by
> "./scripts/build image=java-example,cli" runs the short-lived java-example
> but then the CLI doesn't work: It can't run any commands because the
> httpserver it relies on is already dead.
>
> So this patch partially reverts e270cf9c... We keep the "app registry"
> introduced there, but we still have a separate list of background apps
> started in the command line - and wait for them to finish before asking
> httpserver to exit.
>
> Signed-off-by: Nadav Har'El 

Looks good to me.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] virtio-rng: don't hang while waiting for entropy

[Pekka Enberg]

On Thu, Nov 17, 2016 at 10:39 AM, Nadav Har'El <n...@scylladb.com> wrote:
> random_kthread() has a loop, where it tries to get some entropy from
> all sources (including the ring of internal events), feeds it into the
> entropy pool, and then sleeps a bit and tries again, ad infinitum.
>
> If the read function of *one* of the entropy sources blocks, the entire
> random_kthread() blocks, and NONE of the entropy sources get used - not
> even the internal one. This is bad. For example, if one of the entropy
> sources blocks during boot, none of the plenty of internal random events
> that happen during boot get used, and when the user's application uses
> /dev/urandom on startup, it gets blocked because the entropy pool has not
> been filled.
>
> So the read function of live entropy sources should never block. In fact
> a comment in live_entropy_source_feed() suggests that "This should be
> quick, since it's a live entropy source". An entropy source which cannot
> at this point provide any entropy should return zero bytes, instead of
> blocking until it has some. The caller will find random bytes from other
> sources, and doesn't want to wait for this specific source!
>
> So virtio-rng's read function shouldn't wait for random bytes, it should
> just provide whatever bytes it has right now.
>
> Fixes #805
>
> Signed-off-by: Nadav Har'El <n...@scylladb.com>

The virtio-rng part looks correct:

Reviewed-by: Pekka Enberg <penb...@scylladb.com>

However, perhaps we ought to also fix live_entropy_sources_feed()
_not_ to feed events to entropy_processor() if entropy source read
returned zero like later versions of FreeBSD do:

https://github.com/freebsd/freebsd/blob/master/sys/dev/random/random_harvestq.c#L221

The problem with an event that has no entropy is that the only unique
thing the event has is the cycle count and that's probably also
predictable. So we end up seeding the PRNG with predictable sources in
many cases.

- Pekka

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [COMMIT osv master] Update apps/ submodule

[Pekka Enberg]

On Wed, Sep 14, 2016 at 9:05 AM, Commit Bot  wrote:
> From: Nadav Har'El 
> Committer: Nadav Har'El 
> Branch: master
>
> Update apps/ submodule



>
> Signed-off-by: Nadav Har'El 
>
> ---
> diff --git a/apps b/apps
> --- a/apps
> +++ b/apps
> @@ -1 +1 @@
> -Subproject commit d9cfadc10e80f0ddf87571c026a6f7be669f03fe
> +Subproject commit e3ee7a7b923488a96c90e8b2deb94d5a93a3762a
>
> --
> You received this message because you are subscribed to the Google Groups
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to osv-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] drivers/vmxnet3: Fix driver state issues

[Pekka Enberg]

On Wed, Aug 31, 2016 at 6:13 PM, Timmons C. Player
 wrote:
> This commit fixes two observed issues with the vmxnet3 driver:
>
> * Handle both missing SOP and EOP descriptors in the rx loop. ESXi
>   drops descriptors under load.
> * Always leave at least one Tx descriptor available.  The driver
>   appears to hang when it runs out of them.
>
> Signed-off-by: Timmons C. Player 

Let's CC Takuya who wrote the driver.

> ---
>  drivers/vmxnet3.cc | 13 -
>  1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/vmxnet3.cc b/drivers/vmxnet3.cc
> index 926d1b0..0739303 100644
> --- a/drivers/vmxnet3.cc
> +++ b/drivers/vmxnet3.cc
> @@ -537,7 +537,7 @@ int vmxnet3_txqueue::try_xmit_one_locked(void *req)
>
>  int vmxnet3_txqueue::try_xmit_one_locked(vmxnet3_req *req)
>  {
> -auto count = req->count;
> +auto count = req->count + 1;
>  if (_avail < count) {
>  gc();
>  if (_avail < count)
> @@ -830,6 +830,17 @@ void vmxnet3_rxqueue::receive()
>  goto next;
>  }
>
> +/* Check and handle SOP/EOP state errors */
> +if (rxcd->layout->sop && _m_currpkt_head) {
> +m_freem(_m_currpkt_head);
> +_m_currpkt_head = _m_currpkt_tail = nullptr;
> +stats.rx_drops++;
> +} else if (!rxcd->layout->sop && !_m_currpkt_head) {
> +stats.rx_drops++;
> +discard(rid, idx);
> +goto next;
> +}
> +
>  if (rxcd->layout->sop) {
>  assert(rxd->layout->btype == btype::head);
>  assert((idx % 1) == 0);
> --
> 2.7.4
>
> --
> You received this message because you are subscribed to the Google Groups 
> "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to osv-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] Remove deprecated "mgmt" submodule

[Pekka Enberg]

On Thu, Jun 30, 2016 at 2:29 PM, Nadav Har'El  wrote:
> Pekka or Avi, can I please have your opinion on this patch?
>
> Is it ok with you that we drop the old "mgmt" shell, and stop building
> images for it?

I am certainly fine with that.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.