Re: Exodus: Lightweight relocation engine. Useful?

[Dan Kaminsky]

So Exodus will in fact compile an executable, via dietlibc or musl, but
it's an execve wrapper, and it seems you've stubbed that.

You have to change the relevant line in src/exodus_bundlers/launchers.py to:

args = initial_args + ['-shared', '-fPIC', '-O3', input_filename,
'-o', output_filename]

# ./scripts/run.py
OSv v0.24-488-g06704deb
eth0: 192.168.122.15
execve() stubbed

But at least it parses! :)

I suspect it's statically linking too much for you, and really needs to be
dynamic enough to pull in your libc.  But that might actually be possible.

On Fri, Feb 2, 2018 at 5:17 PM, Rick Payne  wrote:

> On Fri, 2018-02-02 at 16:28 -0800, Dan Kaminsky wrote:
> > That's where I hit my wall, but it seems pretty much a bullseye on
> > your wheelhouse.  Suggestions?
>
> I saw the exodus announcement too - very nice. I wondered if you could
> alter it to emit a C program that dlopen()s the binary, looks up main()
> and calls it? Thats what has been done in several places (for some of
> the erlang 'port' functionality for instance).
>
> The executable and libraries would still need to be pie, but with many
> linux distros using that by default, it may not be so limiting...
>
> Rick
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Exodus: Lightweight relocation engine. Useful?

[Rick Payne]

On Fri, 2018-02-02 at 16:28 -0800, Dan Kaminsky wrote:
> That's where I hit my wall, but it seems pretty much a bullseye on
> your wheelhouse.  Suggestions?

I saw the exodus announcement too - very nice. I wondered if you could
alter it to emit a C program that dlopen()s the binary, looks up main()
and calls it? Thats what has been done in several places (for some of
the erlang 'port' functionality for instance).

The executable and libraries would still need to be pie, but with many
linux distros using that by default, it may not be so limiting...

Rick

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Exodus: Lightweight relocation engine. Useful?

[Dan Kaminsky]

Good news -- some of the code I was hoping to build just came out, and
since I didn't write it, it's actually pretty good :D

https://github.com/Intoli/exodus

or:

pip install exodus-bundler

Run:

exodus ssh -t | tar xz

And you get a self contained binary distributon, for most things short of
maybe browsers.  Launch point is at:

./exodus/bin/ssh

Downside:  You don't get an elf executable.  You get:

#! /bin/bash

current_directory="$(dirname "$(readlink -f "$0")")"
lib_directory="${current_directory}/../lib/"
linker="${lib_directory}/ld-linux-x86-64.so.2"
executable="${current_directory}/ssh"
exec "${linker}" --library-path "${lib_directory}" --inhibit-rpath ""
"${executable}" "$@"

That's where I hit my wall, but it seems pretty much a bullseye on your
wheelhouse.  Suggestions?

--Dan

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Allocating and switching to larger stack during syscall

[Waldek Kozaczuk]

I am trying to implement the idea that Nadav proposed to save memory when 
implementing callstack switch during syscall. Here is excerpt from our 
conversation:

" 

>
> I was also wondering that existing implementation of syscall stack makes 
> every app thread allocate it upon creation eagerly whether syscall are 
> going to be made or not. For example java apps (which I believe use libc 
> functions) would be penalized by memory usage. I was wondering how 
> difficult it would be to make allocation of syscall stack lazily only when 
> first syscall is made.
>

The problem is that we won't have a stack to do this allocation work on.

One thing we could do have for all threads a tiny syscall stack (doesn't 
even need to be a whole page, can be smaller!), and the first thing a 
syscall does is
to check if this thread has this tiny stack, and if it does it allocates a 
different one. The stack needs to be just big enough for malloc() to run. 
We could
even avoid calling malloc() directly and just send a message to a different 
thread to do this for us, but I think this will be an overkill.
"

So I am trying to avoid messing with assembly as much as possible. Is it 
possible to implement it almost all in C by calling a function from 
linux.cc/syscall_wrapper() that would lazily allocate larger stack? 
Obviously we would still need to change RSP register to new value and then 
restore it back with could be done with asm("...") instruction. Is it 
possible?

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH v2] Implemented Read-Only File System (ROFS)

[Waldek Kozaczuk]

This one is identical to previous patch except it states v2 to clearly 
indicate it is 2nd version of the original patch I sent in December. It is 
essentially a minimal subset of ROFS implementation without cache layer. It 
also does not include unit tests.

I will be sending a followup patch to allow unit tests run against ROFS 
image with /tmp mounted as ramfs.

Waldek

On Friday, February 2, 2018 at 8:29:13 AM UTC-5, Waldek Kozaczuk wrote:
>
> The Read-Only File System (ROFS) is a simple implementation of 
> a file system where data from disk can only be read from and never 
> written to. It is simple enough for many stateless applications 
> deployed on OSv which only need to read code from local disk and never 
> write to local disk. The ROFS is inspired and shares some ideas 
> from the original MFS implementation by James Root from 2015. 
>
> This initial implementation of ROFS operates without cache. 
> This means that ROFS reads as much data from disk as requested 
> per uio passed in to the read function but it does not retain/cache it 
> for any subsequent read of the same data. 
>
> The image built with ROFS is automatically set up to mount /tmp 
> as ramfs filesystem to allow writing of any transient data like logs. 
>
> The layout of the data on disk is as follows: 
>
> - Super Block (512 bytes) that contains magic number and specifies meta 
>   information including block size and location and size of tables 
> containing 
>   i-nodes, dentries and symbolic links 
>
> - Files data where each file is padded to 512 bytes block 
>
> - Table of directory entries referenced by index in directory i-node 
>   (each entry holds string with direntry name and i-node number) 
>
> - Table of symlinks referenced by symlink i-node (each entry holds 
> symbolic link 
>   path string) 
>
> - Table of inodes where each specifies type (dir,file,symlink) and data 
> offset 
>   (for files it is a block on a disk, for symlinks and directories it is 
> an 
>   offset in one of the 2 tables above) 
>
> Besides ROFS implementation this patch also changes VFS main to 
> automatically 
> mount ROFS or ZFS. It also adds number of new metrics that are captured 
> and output 
> in verbose mode. 
>
> The images with RFS can be built by specified fs=rofs option like so: 
> ./scripts/build image=node-fs-example -j4 fs=rofs 
> ./scripts/build image=openjdk9-java-base,java-example -j4 fs=rofs 
>
> Potential future improvements to ROFS: 
> - add caching layer to read ahead more data from disk in anticipation 
>   of sequantial access 
> - add compression of file segments 
> - memory page sharing when file mmaping (implement vnop_cache) 
> - add LRU logic to ROFS cache 
>
> Signed-off-by: Waldemar Kozaczuk  
> --- 
>  Makefile   |   4 + 
>  fs/rofs/rofs.hh| 117 +++ 
>  fs/rofs/rofs_common.cc |  85 + 
>  fs/rofs/rofs_vfsops.cc | 216  
>  fs/rofs/rofs_vnops.cc  | 303 
> + 
>  fs/vfs/main.cc |  67 ++-- 
>  fs/vfs/vfs_conf.cc |   3 + 
>  fs/vfs/vfs_vnode.cc|   9 +- 
>  include/osv/vnode.h|   1 + 
>  licenses/mfs.txt   |  46 +++ 
>  loader.cc  |  24 +- 
>  modules/httpserver-api/api/fs.cc   |   4 +- 
>  scripts/build  |  22 +- 
>  scripts/gen-rofs-img.py| 373 
> + 
>  static/etc/fstab_rofs  |   4 + 
>  usr_nozfs.manifest.skel => usr_ramfs.manifest.skel |   0 
>  usr_nozfs.manifest.skel => usr_rofs.manifest.skel  |   2 +- 
>  17 files changed, 1245 insertions(+), 35 deletions(-) 
>  create mode 100644 fs/rofs/rofs.hh 
>  create mode 100644 fs/rofs/rofs_common.cc 
>  create mode 100644 fs/rofs/rofs_vfsops.cc 
>  create mode 100644 fs/rofs/rofs_vnops.cc 
>  create mode 100644 licenses/mfs.txt 
>  create mode 100755 scripts/gen-rofs-img.py 
>  create mode 100644 static/etc/fstab_rofs 
>  copy usr_nozfs.manifest.skel => usr_ramfs.manifest.skel (100%) 
>  rename usr_nozfs.manifest.skel => usr_rofs.manifest.skel (88%) 
>
> diff --git a/Makefile b/Makefile 
> index 8dd2371..2a02a19 100644 
> --- a/Makefile 
> +++ b/Makefile 
> @@ -1769,6 +1769,10 @@ fs_objs += ramfs/ramfs_vfsops.o \ 
>  fs_objs += devfs/devfs_vnops.o \ 
>  devfs/device.o 
>   
> +fs_objs += rofs/rofs_vfsops.o \ 
> +rofs/rofs_vnops.o \ 
> +rofs/rofs_common.o 
> + 
>  fs_objs += procfs/procfs_vnops.o 
>   
>  objects += $(addprefix fs/, $(fs_objs)) 
> diff --git a/fs/rofs/rofs.hh b/fs/rofs/rofs.hh 
> new file mode 100644 
> index 000..9fcd9a9 
> --- /dev/null 
> +++ b/fs/rofs/rofs.hh 
> @@ -0,0 +1,117 @@ 

[PATCH v2] Implemented Read-Only File System (ROFS)

[Waldemar Kozaczuk]

The Read-Only File System (ROFS) is a simple implementation of
a file system where data from disk can only be read from and never
written to. It is simple enough for many stateless applications
deployed on OSv which only need to read code from local disk and never
write to local disk. The ROFS is inspired and shares some ideas
from the original MFS implementation by James Root from 2015.

This initial implementation of ROFS operates without cache.
This means that ROFS reads as much data from disk as requested
per uio passed in to the read function but it does not retain/cache it
for any subsequent read of the same data. 

The image built with ROFS is automatically set up to mount /tmp
as ramfs filesystem to allow writing of any transient data like logs.

The layout of the data on disk is as follows:

- Super Block (512 bytes) that contains magic number and specifies meta
  information including block size and location and size of tables containing
  i-nodes, dentries and symbolic links

- Files data where each file is padded to 512 bytes block

- Table of directory entries referenced by index in directory i-node
  (each entry holds string with direntry name and i-node number)

- Table of symlinks referenced by symlink i-node (each entry holds symbolic link
  path string)

- Table of inodes where each specifies type (dir,file,symlink) and data offset
  (for files it is a block on a disk, for symlinks and directories it is an
  offset in one of the 2 tables above)

Besides ROFS implementation this patch also changes VFS main to automatically
mount ROFS or ZFS. It also adds number of new metrics that are captured and 
output
in verbose mode. 

The images with RFS can be built by specified fs=rofs option like so:
./scripts/build image=node-fs-example -j4 fs=rofs
./scripts/build image=openjdk9-java-base,java-example -j4 fs=rofs

Potential future improvements to ROFS:
- add caching layer to read ahead more data from disk in anticipation 
  of sequantial access
- add compression of file segments 
- memory page sharing when file mmaping (implement vnop_cache)
- add LRU logic to ROFS cache

Signed-off-by: Waldemar Kozaczuk 
---
 Makefile   |   4 +
 fs/rofs/rofs.hh| 117 +++
 fs/rofs/rofs_common.cc |  85 +
 fs/rofs/rofs_vfsops.cc | 216 
 fs/rofs/rofs_vnops.cc  | 303 +
 fs/vfs/main.cc |  67 ++--
 fs/vfs/vfs_conf.cc |   3 +
 fs/vfs/vfs_vnode.cc|   9 +-
 include/osv/vnode.h|   1 +
 licenses/mfs.txt   |  46 +++
 loader.cc  |  24 +-
 modules/httpserver-api/api/fs.cc   |   4 +-
 scripts/build  |  22 +-
 scripts/gen-rofs-img.py| 373 +
 static/etc/fstab_rofs  |   4 +
 usr_nozfs.manifest.skel => usr_ramfs.manifest.skel |   0
 usr_nozfs.manifest.skel => usr_rofs.manifest.skel  |   2 +-
 17 files changed, 1245 insertions(+), 35 deletions(-)
 create mode 100644 fs/rofs/rofs.hh
 create mode 100644 fs/rofs/rofs_common.cc
 create mode 100644 fs/rofs/rofs_vfsops.cc
 create mode 100644 fs/rofs/rofs_vnops.cc
 create mode 100644 licenses/mfs.txt
 create mode 100755 scripts/gen-rofs-img.py
 create mode 100644 static/etc/fstab_rofs
 copy usr_nozfs.manifest.skel => usr_ramfs.manifest.skel (100%)
 rename usr_nozfs.manifest.skel => usr_rofs.manifest.skel (88%)

diff --git a/Makefile b/Makefile
index 8dd2371..2a02a19 100644
--- a/Makefile
+++ b/Makefile
@@ -1769,6 +1769,10 @@ fs_objs += ramfs/ramfs_vfsops.o \
 fs_objs += devfs/devfs_vnops.o \
devfs/device.o
 
+fs_objs += rofs/rofs_vfsops.o \
+   rofs/rofs_vnops.o \
+   rofs/rofs_common.o
+
 fs_objs += procfs/procfs_vnops.o
 
 objects += $(addprefix fs/, $(fs_objs))
diff --git a/fs/rofs/rofs.hh b/fs/rofs/rofs.hh
new file mode 100644
index 000..9fcd9a9
--- /dev/null
+++ b/fs/rofs/rofs.hh
@@ -0,0 +1,117 @@
+/*
+ * Copyright (c) 2015 Carnegie Mellon University.
+ * All Rights Reserved.
+ *
+ * THIS SOFTWARE IS PROVIDED "AS IS," WITH NO WARRANTIES WHATSOEVER. CARNEGIE
+ * MELLON UNIVERSITY EXPRESSLY DISCLAIMS TO THE FULLEST EXTENT PERMITTEDBY LAW
+ * ALL EXPRESS, IMPLIED, AND STATUTORY WARRANTIES, INCLUDING, WITHOUT
+ * LIMITATION, THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+ * PURPOSE, AND NON-INFRINGEMENT OF PROPRIETARY RIGHTS.
+ *
+ * Released under a modified BSD license. For full terms, please see mfs.txt in
+ * the licenses folder or contact permi...@sei.cmu.edu.
+ *
+ * DM-0002621
+ *
+ * Based on https://github.com/jdroot/mfs
+ *
+ * Copyright (C) 2017 Waldemar Kozaczuk
+ * Inspired by original MFS