Hi Erin,
In Cloudflare Workers, we've had some success attaching certain kinds of
metadata by having a wrapper object which is called first, like:
interface Endpoint {
startCall @0 (metadata :Metadata) -> (dispatcher :CallDispatcher);
}
interface CallDispatcher {
foo @0 (...) -> (...);
bar @1 (...) -> (...);
...
}
Here, the client always first calls startCall() to pass the metadata, then
makes a pipelined call to the specific method. By convention, a
`CallDispatcher` is expected to be used for exactly one call.
Of course, there's a limit to how far this pattern can go. It has worked
well for us so far but it's definitely a bit clunky.
I'm really hesitant to attach a totally untyped int -> any map to every
call. Having built several systems using such a pattern in the past, I know
it can be very useful, but also tends to turn into a mess in certain ways.
For the specific use case of tracing, I can see a strong argument that this
should be built into the system. You really do want trace IDs to pass down
the call stack implicitly, without the business logic having to think about
it (except in cases where the business logic wishes to explicitly declare a
span). To make this work well in Cap'n Proto C++, we'd want to extend KJ's
promise framework to implicitly propagate a trace context along the promise
chain, so that the RPC system can pick it up.
I think authentication is at the opposite end of the spectrum. Implicitly
propagating authentication credentials goes against the capability
philosophy and gives rise to confused deputy problems. Authentication
should be achieved by having the client exchange their credentials for an
authenticated capability upfront, and then use that capability for further
calls. A membrane should revoke the capability if the credentials become
invalid. So I would not want to create a system that might be abused for
authentication.
Lamport timers are interesting. I don't have a lot of intuition here as I
haven't worked with them much. Naively it seems like you want to maintain a
time counter in each vat, send the vector with each call/return, and merge
the vectors received... however, this implies that the timer is ambient
mutable state, which introduces lots of problems. Does it become a side
channel through which it's possible to observe what else is going on within
the vat? This makes me a bit uncomfortable.
Maybe you could implement Lamport clocks with membranes. You could wrap
each group of objects that share a time counter in a single membrane. When
a call exits the membrane, the membrane could first generate a preceding
call to the same target object to some common propagateLamportTime()
method. On the receiving end, the membrane intercepts this call and updates
the receiver's clock. E-order guarantees that the propagateLamportTime()
call is received immediately before the call that caused it. Alternatively,
you could actually wrap outgoing calls in an envelope that include the
timestamp, and have the receiving membrane unwrap the envelope.
-Kenton
On Mon, Oct 12, 2020 at 1:15 PM Erin Shepherd <[email protected]> wrote:
> Many protocols provide some way to attach metadata to requests &
> responses. HTTP and protocols derved from it provide this in the form of
> headers (and occasionally trailers), for example.
>
> There are various reasons why this can be very useful, even in a
> capability oriented system:
>
> - Propagating a request correlation ID from node to node as a call
> passes through multiple parts of a system for logging correlation purposes
> - Even better, propagating an OpenTracing SpanContext or similar to
> enable correlating multiple parts of an action in a distributed system with
> tools like Jaeger <https://www.jaegertracing.io/>
> - Attaching Lamport timestamps
> <https://en.wikipedia.org/wiki/Lamport_timestamp> to every request and
> reply, to propagate an inter-system causal ordering
>
> These sorts of things are very useful to convey *implicitly*. When every
> node in your system deals in lamport timestamps, then manually annotating
> every call with a timestamp parameter & manually handling that parameter
> and adding it to the response will quickly get tedious. If you want to do
> distributed tracing, then you want that context everywhere.
>
> These kinds of things represent the kind of state/context which is
> implicit from the callstack in traditional single-threaded applications
>
> There are some other places where I think metadata could be useful,
> although they probably are also more matters of taste, like
> identity/authentication (one of my thoughts here is that sometimes it might
> be advantageous to use metadata to pass this kind of information because
> this ensures it is always in a uniform location where intermediate
> proxies/membranes can find it without needing to be aware of involved
> schemas. You could potentially see a system which implements
> authentication/authorisation through collaborating membranes where those
> membranes need to communicate identifying information with each other, for
> example.)
>
> There are some questions that would need resolving:
>
> - How should metadata be represented? (I'd lean towards something like
> a Map<UInt64 TypeID, AnyPointer assumed to be of TypeID>)
> - What messages can it apply to? Call/Return are obvious, but there
> are other possibilities that might be necessary (Embargo/Disembargo?
> Provide/Accept? Join?). Maybe metadata should just be a paramete of the
> top-level message?
> - How to access/manipulate it? Presumably in C++ this could be exposed
> via CallContext on the server side / Request on the client side?
> - How to support "middleware" and/or pass such ambient context around?
> I feel the potentially "easy" option is for middleware to simply be a
> membrane, but ambient context will depend upon language. In Go you might
> use context.Context for example; not sure what you would do in e.g. KJ
>
> I realise this probably opens several cans of worns, and that metadata
> tends to always be somewhat inelegant, but it seems worthwhile to talk
> about becuase it's important in lots of distributed systems scenarios.
>
> - Erin
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/9ccbf8e1-b013-4744-be5f-d7b789bfa555%40www.fastmail.com
> <https://groups.google.com/d/msgid/capnproto/9ccbf8e1-b013-4744-be5f-d7b789bfa555%40www.fastmail.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D7or__QqJ3SKk8Bi9eyXX3oLXGEVQtCmZAefihJ6hBmQ%40mail.gmail.com.
