On 08/11/15 18:00, Gabriel Parmer wrote:
Composite is a component-based OS similar to a micro-kernel in which
policies are implemented as user-level, isolated, components,
interconnected with fast IPC. Composite focuses on predictable execution
(i.e. bounded-latency for real-time systems), while also scaling that
predictability up with an increasing number of features.
Now for a bonus run-on sentence that shows what we've been using the system
for. A few outrageous (but substantiated) claims: the kernel has no
scheduling policy, instead supporting system-wide scheduling in a set of
user-level components; the kernel has no locks but maintains consistency
while scaling perfectly (we've tested up to 40 cores on 4 sockets); the
kernel has no dynamic memory allocator, instead _safely_ relying on
user-level to perform that function; system level functionalities (e.g.
scheduler, memory mapper) can fail, be rebooted, and reintegrated into the
system both predictably, and efficiently (in 10s of microseconds); and a
custom web-server is implemented as a system composed of >25 different
components.
Thanks. I still wasn't quite sure what the raison d'etre of Composite
is. So I went digging and found this:
https://www.seas.gwu.edu/~gparmer/publications/ospert13_cos.pdf
(I'll assume that's from 2013)
At least now I know why it's called "Composite".
Hmm, I can't imagine how a thread scheduler would fail so that you can
detect it and reboot the service.
RK fits into our plans as 1. it will give us access to quite a few device
drivers, 2. it provides some degree of instant legacy support, 3. we'd like
to use our IPC support to provide an efficient rumpctl equivalent, and 4.
we'd ideally like to run multiple RKs, each configured with different
subsets of overall system functionality.
But a rump kernel won't be useful for anything where you want to
guarantee bound latency, no? Or does a tool exist which can figure
things out? Sounds pretty difficult, even with annotations.
If by "3" you mean you'd like to run sysproxy (as described in chapter
3.12.4 of the blue book) on top of your IPC, sure. The current POSIX
implementation is unfortunately incorrect abstracted on top of sockets.
Though, if your endpoints are not running on top of POSIX, it doesn't
really matter, since you need to write your own implementation anyway.
The sysproxy protocol is expected to be stable. Though, I want to add
one more message type to signal "configuration complete", but never seem
to get around to it. There are a bunch of parties interested in getting
alternative sysproxy transports going. Hopefully there can be maximal
sharing and minimal reinventing of the wheel (which is not to say that
nobody needs to do any work ;)
I like "4".
In the new terminology world order, bmk is the MI+MD "OS" which supports
both hardware and Xen.
I must admit that I haven't had time to follow this whole discussion on the
list (the past month has been busy). Is there documentation for the MI+MD
in the NWO?
No documentation really. "bmk" (bare metal kernel) used to be the name
of the hw platform, but when rumprun-xen and rumprun-baremetal were
unified, bmk was repurposed as the name for the the low-level bits in
terms of which rumpuser is implemented. If you look into rumprun/lib,
it should make things clearer.
If you can get the compiler/linked to generate __tls_get_addr() for TLS
access, AFAIU you can just change bmk_platform_cpu_sched_settls() to do
whatever corresponds to your __tls_get_addr()?
Quite a few of our struggles so far have involved understanding the build
system and the system's configuration. We are not NetBSD users (though RK
is pushing us in that direction).
The higher order bit we'd like to understand -- and I think I know the
answer now -- is if there are configuration options with RK and within
NetBSD environment to control the implementation of TLS. I'm getting the
impression that all of that responsibility is punted out to the host's
compiler, and the RK's code uses what is generated by the compiler in the
standard way. Thus, aside from bmk's configuration option which only
impacts bmk code, there is no broader configuration to change the
implementation of TLS.
Again, there is no use of __thread in the rump kernel. The only place
where __thread can optionally be used is for curlwp, as documented here:
http://nxr.netbsd.org/xref/src/sys/rump/README.compileopts?r=1.13#58
The default, as you'll note, is hypercall. That said, Rumprun uses
__thread, because it would be wasteful not to. Again, Rumprun is not a
framework but an end result, so it doesn't have to take into account all
problems anyone might ever have.
I'm still not 100% sure of what kind of change you propose to bmk. What's
confusing me is that you've been porting and changing various bits, but now
don't want to change various other bits.
If the problem were only in bmk, then we'd have no issue. We can modify
that without significant impact on the rest of the system. However, we'd
like like to avoid modifying the rest of the system to the greatest extent
possible. What triggered this email was seeing that the NetBSD libc uses
__thread for errno (which is perfectly logical). We can also "fix" that
reference, but before making broader changes outside of the
platform-specific code, we wanted to see if anyone knew of any other option.
First of all, the NetBSD libc used in Rumprun does not include the
standard implementation of errno -- see use of ".if ${RUMPRUN}" in the
libc Makefile. As far as I can recall, there are two reasons: 1) errno
was needed before a bunch of other stuff was figured out 2)
https://github.com/rumpkernel/rumprun/blob/61e5b8d98bd8a3f665f030e4935d604ba90d11ff/lib/libbmk_core/sched.c#L729-L739
(ok, technically we could just use a compile-time definition for
kernonly-mode to solve "2", but since what was done for "1" continues to
work, there has been no reason to change it)
But, if you're using libc, you probably want some real applications to
run on top, and you can't expect none of those to use __thread, so
eliminating __thread from system components seems really like a non-starter.
So me just rephrase some things once more. The Rumprun unikernel is
made up of multiple layers, only one of which is the rump kernel. The
logical stack of layers looks something like this. Let's called this
stack "1":
POSIX application (including libs)
libc + rumprun base
rump kernel
bmk (including rumpuser)
Or, alternatively, if you're shooting for minimal or simply don't care
about existing POSIX software, you can use "kernonly" Rumprun and write
your application from scratch against the rump kernel syscall
interfaces, stack "2":
homegrown application
rump kernel
bmk (including rumpuser)
And, though I think you already said you evaluated the case without bmk
and your own implementation of rumpuser, for the sake of completeness:
you can swap out bmk from the above and still have a working system.
However, if you want to replace bmk in "1", you need to note that bmk is
used to implement some functionality used by libc due to the rump kernel
not providing said functionality (e.g. page allocator). So, if you're
interested in your own bmk implementation for stack 1, you need to be
mindful. For stack 2 you only have to worry about the rumpuser
interface ... because, once again, the rump kernel was designed and
implemented as an integratable framework with no standalone value, while
Rumprun was designed and implemented as a "full stack" solution.
