Thanks for the rather lengthy overview (which is quoted below -- I'm not
responding to any particular point, but since it's been a while since
the original post, I want to avoid folks having to go hunting in the
archives to refresh their memories).
I'd like to summarize your mail as two points:
1) the current problem with the rumprun stack in userspace is that there
is no perfect way to work around two copies of libc in the same process.
frankenlibc addresses the problem by providing the bare minimum bits
to avoid having to link in the host libc at all.
2) rumprun-posix currently provides both "local" and "remote" clients,
the use cases of which are completely different, in addition to the
underlying implementations requiring different things. Part of the
implication of your work is to split local and remote into separate
pieces, as they should logically be. You want to initially use
frankenlibc only for the local version.
Hopefully you find the summary accurate.
Then some comments:
I think frankenlibc will fit very nicely into the unified rumprun (xen +
bare metal) repo as a platform. I think I'll be able to unify most of
the rumpuser implementation in there, apart from I/O calls. It probably
makes sense for a frankenlibc rumprun stack to use that rumpuser
implementation as opposed to the POSIX one, especially since frankenlibc
rumprun decidedly isn't POSIX-based.
I am guessing that frankenlibc will be valuable also for a better
implementation of remote clients. Sure, the "hypercalls" that
librumpclient runs on are different from the ones that the rump kernel
uses, but the collision problem with the host libc is the same, as is
the requirement to "boot" the hosting process and make syscalls -- just
slightly different syscalls than for the local case. Do you think we
should copy the bits, use a subrepo, or something else? I don't
immediately see a perfect solution for this one.
The user-visible impact is that the remote clients for controlling rump
kernels via sysproxy will move repo (or at least the repo name will
change). This is actually quite a biggie. I think it deserves a
separate heads up mail with a clear subject ;)
All in all, very cool work!
- antti
p.s. is frankenlibc the final word for supporting the rumprun stack in
userspace, or do you have yet another improved plan hidden somewhere? ;)
On 05/03/15 12:45, Justin Cormack wrote:
It has become clear that there is some confusion about what
frankenlibc is for and whats going on with it, so here is a bit of
explanation.
Note that what is currently called "rumprun-posix" (the original
rumprun repo) does two things at present, in similar ways. These are
providing NetBSD binaries (and compiler wrapper) for binaries that
talk to a remote rump kernel. We will refer to thse as "remote". Also
it provides binaries and a link shell script for binaries including
their own rump kernel, which we will call "local".
The fact that it does two different things is pretty confusing form
the point of view of documentation. The most useful thing is the
remote commands, which can also talk to Xen kernels now, and is the
main way to configure rump kernels for debugging etc.
The local versions of the commands are basically useless, as the
kernel shuts down as soon as the config is applied. This model is
basically useful for custom applications rather than existing
commands, as the userspace version of the Xen, baremetal repos. This
is the space that frankenlibc is aimed at, so I am proposing removing
the local stuff from the rumprun-posix repo, and suggest renaming it
something referring to "remote", eg "rumprun-remote".
Rumprun-posix works by renaming symbols in NetBSD libc and the
marching symbols in the user code so they no longer clash with the
host libc, localizing some symbols so they are invisible, subverting
the startup and shutdown process by renaming the user's main and exit.
The process is pretty complex, see
https://github.com/rumpkernel/rumprun-posix/blob/master/mkrun.sh and
while it could be made into a gcc ld script, it is not devoid of
hacks. It is also another reason why having two versions in the repo
for local and remote makes it harder to maintain.
Frankenlibc is a complete rewrite from scratch. It allows the user
program and NetBSD libc to own their namespace, main() and so on. To
understand how its built it helps to understand the way a rump kernel
links. The majority of the rump kernel, librump and librump* libraries
are all namespaced as rumpns_ and do not conflict with anything.
librump when compiled with the rump-is-libc option provides the
syscall functions that the NetBSD libc expects. The bit that causes
issues is librumpuser, which needs to talk to the host, so would
normally call into host libc.
Frankenlibc has its own librumpuser, which is essentially a version of
the fiber implementation from NetBSD. That still needs some libc
functions of course although not many, so frankenlibc provides its own
implementations of these, namespaced through #defines in its headers.
This is all portable code, other than the architecture dependent
swapcontext implementation used for fibers which is written in
assembly with x86_64 and i386 code currently. Much of the libc code
comes from Khorben's DeforaOS libc, with fixes, and the malloc is
currently the same one we use for Xen and baremetal. In theory it
could be pulled from NetBSD libc, and in particular new versions of
jemalloc support not using brk just page allocations, so it would be
sane to use that.
Obviously at some point it then needs platform features, although as
the rump hypercall interface is small it doesnt need many. The main
one is it needs a page allocator so mmap can work, and it helps to
have console output. For the currently supported platforms, Linux,
NetBSD and FreeBSD these use direct system calls in assembly. However
you could write them in C if your platform has an aligned memory
allocator that does not clash with the libc namespace, eg RTEMS has
rtems_memalign which should work fine. As you can see, the platform
support is not "posix" at all, and Posix platforms are treated like
non Posix ones, except they are slightly harder to write.
In addition to the hypercall implementation layer, there is some
startup code. This is split into platform specific startup, and then a
general portable entry point which initialises rump, NetBSD libc,
calls any init code, and then calls main(). For the current platforms
there is a per platform crt0.o/crt1.o. This is what allows you to just
use a standard unmodified C compiler so long as you also link in the
frankenlibc at the end to provide the symbols that librumpuser uses.
eg ${CC} ${CFLAGS} ${LDFLAGS} -static -nostdinc -Brump/lib
-Irump/include -Lrump/lib test/hello.c -lc ${RUMP_LDLIBS} -lfranken -o
${TESTDIR}/test - this works with gcc and clang. I will probably fix
it to support a sysroot.
So the architecture is completely different from rumprun-posix, and
you have to write code for each platform. However it actually works on
more platforms than rumprun-posix ever did already. It has some stuff
missing, as I have been tidying up code, so it is still missing
pthread support, will add that back soon. It has had less testing with
real code so far. It only targets static linking, and fibers, so it is
single process, unthreaded.
The aim is to provide an equivalent to the Xen platform that runs in
userspace, so people can run their code there, where it is more
accessible and has better debugging tools. It is also a good starting
point to make a fiber-based port for any other platform as most of the
work has already been done for you, and you should be able to get
started just by adding a memory allocator.
It is also designed to be sandboxed if required, as the system call
set is very well defined, so there will shortly by a Linux seccomp
filtering tool and a Capsicum based tool for FreeBSD. Other current
work is adding device driver support, currently there are etfs block
devices, but obviously it needs network device support - the existing
ones require pthreads so I am adding new ones. Arm support will also
come shortly.
Hope that is a bit clearer.
