On 11/11/15 05:55, Gabriel Parmer wrote:
But a rump kernel won't be useful for anything where you want to guarantee
bound latency, no? Or does a tool exist which can figure things out?
Sounds pretty difficult, even with annotations.
Correct. As it happens, most of the parts of the system that require
strict bounds do not require loads of functionality, and we have facilities
for strictly isolating the wild-west POSIX side from these predictable
parts. I would like to investigate using only the device drivers from RK,
and hooking them with a surrounding predictable core, and seeing what
happens.
I don't think you can say anything conclusive about even just the device
drivers. It's of course easy to see what happens for some set of
events, and even make it deterministic, but that, as I'm sure you know,
won't tell you anything about the situation when various substances hit
the fan.
I honestly haven't looked into the code enough to know how much we'll have
to adapt sysproxy. Our endpoints look identical to function invocation
(but with a round-trip overhead of ~600 cycles on x86-32), and I'd like to
eventually modify sysproxy (with the requisite changes to APIs above and
below) to support our zero-copy message passing.
You can't really do zero-copy without an extensive rework. System call
driver code is generally something like this:
fun(int *userptr)
{
int v;
copyin(userptr, &v);
do_stuff(v);
}
But as long as you can peek and poke the client's memory from the
server, you can at least eliminate the RTT, which is the big cost.
(handwaved around the subject in fig 3.37 of book)
I like "4".
Me too. I see breaking the RK functionality into separate, isolated
components as a logical conclusion.
Can you expand on that thought?
I work in the research domain, so I'm fine with hacking applications to use
our own support for TLS. That said, we've moved on to both 1. manually
changing the offending code that uses __thread to make progress, while also
2. trying to hack in to Composite sufficient, but not complete, support for
using %gs. 1. is motivated by the fact that we'd like to not wait till 2.
is done while still making progress on booting the system.
If you can come up with a patch for bmk which
1) solves your problem
2) incurs no runtime penalty for the current case
3) does not look like civet coffee before the cleanup process
I see no reason not to integrate such a patch.
Some context from our side. Our current implementation plan is as follows:
1. Create a layer below bmk which removes the lowest-level functions, and
implements them using Composite primitives (context switch, initial memory
image allocation, interrupts). We're testing this now, and it has been in
place for the past few months.
So more or less rumprun/platform/$x? (not exactly, but just trying to
locate the ballpark)
2. Get the rumprun unikernel booting with trivial POSIX test programs and
no real devices. We're currently at the point where libc pthread data is
being initialized.
Hmm, libpthread also uses TLS.
3. Get PCI working in the system by implementing the rumpuser interface for
PCI.
.. which reminds me I need to separate the DMA/bus hypercalls from the
PCI components for the benefit of ARM-based system which want to do
device access but don't have PCI.
[snip]
Interesting.
9. Look into directly communicating with the rump kernel (avoiding libc as
in your stack "2" above), and see if we can support a. zero-copy
communication, and b. direct communication with the lower layers of the
rump kernel (i.e. talk almost directly to drivers).
Notably, once you plug into something below the syscall API, you're no
longer in the realm of backward-compatible, stable interfaces --
something to keep in mind.
If you look at libp2k, you'll see one way of plugging in to some
lower-level interfaces. I'm not saying it's the best way, not saying I
can immediately think of a better way either, but at least it's one way.
There are a bunch of ad-hoc solutions for bypassing the syscall layer,
but no real general solution which automatically takes care of
prototypes and data types. Some years ago I tried to get the NetBSD
interface definitions written in xml (because it was the s-expressions
du jour back then) and .h autogenerated from that, but people who
completely missed the point kept screaming "ewww, xml". Oh well.
Also notable: once you move below the syscall drivers, you can't support
remote clients.
Thanks again for all the context and explanations. They have been
informative, and have been an amazingly useful consistency check that our
understanding is mostly in line with reality.
Great, thanks for sharing your plans!
