+1 for non-normative documentation. On Jul 19, 2017 9:54 PM, "Ross Light" <[email protected]> wrote:
I sent #522 <https://github.com/capnproto/capnproto/pull/522> to fix the parts we identified as wrong. So while most of the implications in here can be inferred from the spec, I can understand why they are not part of the spec. What might be good is to have a non-normative comment section that's an "Implementer's Guide" that spells out some of these pitfalls. The guidance from this old thread <https://groups.google.com/d/topic/capnproto/4SLfibQPWFE/discussion> is definitely not obvious, but is implied by other parts of the spec. I'm struggling with some of those implications now, but I'll start a separate thread. On Wed, Jul 19, 2017 at 9:19 PM Kenton Varda <[email protected]> wrote: > On Wed, Jul 19, 2017 at 2:57 PM, Ross Light <[email protected]> wrote: > >> Replies inline (with the disclaimer that I'm not Kenton, my only >> credentials are that I have stared at this file for a long time): >> >> On Wed, Jul 19, 2017 at 1:46 PM Thomas Leonard <[email protected]> wrote: >> >>> Hi, >>> >>> I'm trying to write an implementation of the RPC spec (level 1, in >>> OCaml). I found a few parts of the spec unclear - could someone clarify >>> them for me? >>> >>> It says: >>> >>> [ExportId] >>> > The exporter chooses an ID before sending a capability over the wire. >>> If >>> > the capability is already in the table, the exporter should reuse the >>> same ID. >>> >>> But later: >>> >>> [CapDescriptor] >>> > senderHosted @1 :ExportId; >>> > A capability newly exported by the sender. This is the ID of the new >>> capability in the >>> > sender's export table (receiver's import table). >>> >>> How can the exporter reuse the same ID, if it has to be newly exported? >>> >> >> That seems like a doc/spec typo. You can always specify an existing >> capability. I think the wording should be something like: "A capability >> exported by the sender. This may or may not be a new ID in the sender's >> export table (receiver's import table)." >> > > Correct. > > >> >> >>> [Message] >>> > This could be e.g. because the sender received an invalid or >>> nonsensical >>> > message (`isCallersFault` is true) or because the sender had an >>> internal error >>> > (`isCallersFault` is false). >>> >>> isCallersFault appears to be deprecated (`obsoleteIsCallersFault` >>> appears much later). >>> >> >> Yup, Exception has changed (IMO for the better). Instead of placing >> blame on sender or receiver (such distinctions are hard to draw in >> general), exceptions are now about what action that caller is advised to >> take based on the failure. >> > > Correct. > > >> >> [Call.sendResultsTo] >>> > When `yourself` is used, the receiver must still send a `Return` for >>> the call, but sets the >>> > field `resultsSentElsewhere` in that `Return` rather than including >>> the results. >>> >>> When should `resultsSentElsewhere` be returned? Once the result is >>> known? Or >>> once the first takeFromOtherQuestion collects it? >>> >> >> (I haven't implemented this for Go yet, but want to.) AFAICT >> resultsSentElsewhere should be sent once the result is known. >> > > I think the answer here is "it doesn't really matter". > > When Alice calls Bob.foo(), and Bob tail-calls back to Alice.bar(), Bob > sends the Call to bar() with "send to yourself" *immediately* followed by > the Return for foo() with "take from other question". Eventually Alice > sends a Return for bar(), but Bob doesn't really do anything with this > Return, so it actually doesn't matter when it is sent. That said, the C++ > implementation appears to wait for bar() to finish before sending the > Return. > > >> >> >>> Can takeFromOtherQuestion be used more than once for a single source >>> question? >>> >> >> I would assume that it could be used until Finish message is sent for >> that question, much like other question-based data. In practice, every >> call's result is held in the answers table until Finish is received. >> > > No, it can only be used once. > > For languages without garbage collection, it would be annoying for the > protocol to specify that some messages can potentially be shared. > > >> >> >>> > The `Call` for bar'() has `sendResultsTo` set to `yourself`, with the >>> value being the >>> > question ID originally assigned to the bar() call. >>> >>> What does "the value" refer to here? `yourself` has type `Void`. >>> >> >>> > Vat B receives the `Return` for bar'() and sends a `Return` for bar(), >>> with >>> > `receivedFromYourself` set in place of the results. >>> >>> `receivedFromYourself` does not appear anywhere else in the spec. >>> >> >> I think this whole example is stale and probably needs another draft. >> > > Yeah. I must have had an earlier version where the child call specified > its parent, rather than the parent return specifying the child. > > >> >>> [Return.releaseParamCaps] >>> > If true, all capabilities that were in the params should be considered >>> released. >>> >>> Just to be sure: as if the sender had sent a release message for each >>> one with `count=1`? >>> >> >> (I might be wrong on this point, it's been a while since I've looked. >> The docs should probably spell this out.) Usually. The list of >> CapDescriptors in a Payload could point to the same capability multiple >> times. A release message of count=1 per CapDescriptor is a more accurate >> way of phrasing this. >> > > Correct. > > >> >> >>> [Payload] >>> Why is it not possible to send exceptions in payloads? Should I export >>> each >>> broken capability as an export and then immediately send a Resolve for >>> each >>> one, resolving it to an exception? >>> >> >> Payload is only used for parameters and results. It doesn't make sense >> for parameters to be an exception, and results is inside a union where you >> could specify an exception that is an alternative. I'm not sure I >> understand the use-case where you are sending a broken capability. >> > > Correct that Payload isn't relevant to resolving capabilities. > > To the original question: Hmm, I guess it would have made sense for > CapDescriptor to have an additional variant for an already-broken > capability, to avoid the need for an extra Resolve. > > However, yes, in practice, the C++ implementation will introduce a > promise-capability and then immediately send a Resolve resolving it to an > exception. > > >> >> >>> [Resolve] >>> > When an export ID sent over the wire (e.g. in a `CapDescriptor`) is >>> indicated to be a promise, >>> > this indicates that the sender will follow up at some point with a >>> `Resolve` message. If the >>> > same `promiseId` is sent again before `Resolve`, still only one >>> `Resolve` is sent. If the >>> > same ID is sent again later _after_ a `Resolve`, it can only be >>> because the export's >>> > reference count hit zero in the meantime and the ID was re-assigned to >>> a new export, therefore >>> > this later promise does _not_ correspond to the earlier `Resolve`. >>> >>> It's not clear to me why it is useful for the receiver to know this. >>> Presumably the sender can't reuse an export ID until the receiver >>> explicitly releases it anyway. >>> Should an implementation keep track of whether a resolve has arrived yet >>> and behave differently based on this when it sees an export ID? >>> >> >> It's more specifying that the receiver should not resolve the promise >> more than once. I believe in this case that it would be a protocol >> violation, in which case the correct behavior would be for the receiver to >> send an abort. >> > > The text here is me trying to prove that it's safe for the protocol to > specify that only one Resolve message is sent no matter how many times the > export ID was introduced. I'm trying to show that there's no race > conditions caused by messages travelling in opposite directions passing > each other in-flight. > > Specifically, the rule I'm stating is: After you send a Resolve message > for a promise, you *cannot* attempt to reference the same promise again in > a subsequent message. The associated export ID is off-limits until it has > been released. Once released, it can be reused as normal. Put another way, > after a Resolve, the export ID's refcount is only allowed to decrease until > it hits zero, and then it can increase again. > > The reason for this is that if you referenced an already-resolved promise, > it's possible that the other end has already released the import and sent a > Release message before it receives the new reference. In that case, when it > receives the new reference, it will mistakenly believe that it's hearing > about an all-new promise, and will expect a new Resolve message, which will > never come. > > Luckily, there's no need to reference a promise again once it has been > resolved. It always makes sense to reference the object it resolved to > instead. However, to get this right may require some bookkeeping in the > implementation. > > (If not for this rule, then I would have had to make a different rule > instead: I would have had to say that if you reference a promise again > after resolving it, then you need to send another Resolve message. But that > would have just been wasteful.) > > >> >> > The sender promises that from this point forth, until `promiseId` is >>> released, it shall >>> > simply forward all messages to the capability designated by `cap`. >>> >>> Does something similar apply to Return messages? Might be worth >>> mentioning it there too. >>> >> >> I believe so, but I don't know/remember. :( >> > > The comment here is referring to a rule that resolves the Tribble 4-way > Race Condition, which relates to embargoes, which I'll discuss below. > > The rule says that once you've declared that promise P resolves to > capability C, then any future message address to promise P shall be > forwarded to capability C -- *even if* capability C itself turns out to be > a promise which resolves to D. Even after the resolution, messages to P > cannot be forwarded directly to D -- they *must* be forwarded to C (which > will then forward to D). > > A similar requirement applies to returns, yes. > > >> [Disembargo] >>> > Embargos are used to enforce E-order in the presence of promise >>> resolution. That is, if an >>> > application makes two calls foo() and bar() on the same capability >>> reference, in that order, >>> > the calls should be delivered in the order in which they were made. >>> But if foo() is called >>> > on a promise, and that promise happens to resolve before bar() is >>> called, then the two calls >>> > may travel different paths over the network, and thus could arrive in >>> the wrong order. In >>> > this case, the call to `bar()` must be embargoed, and a `Disembargo` >>> message must be sent along >>> > the same path as `foo()` to ensure that the `Disembargo` arrives after >>> `foo()`. >>> >>> What does "this case" refer to? When exactly is an embargo needed, and >>> when not? >>> >> >> If you're implementing level 1 (two-party), then really the only place >> where this applies is when you receive a capability that the receiver hosts >> as part of a return or resolve after you have made calls on the promised >> capability. This implies that the RPC system needs to keep track of which >> parts of the answer have had calls made on them. When this occurs, the >> receiver gives the application code an embargoed client, and then sends a >> Disembargo with senderLoopback set. It releases the embargo once the same >> disembargo ID is returned with receiverLoopback set. >> >> For me, this was the hardest part of the spec to understand. I >> understand why it's needed, but it's really hard to grok the implications. >> > > Example: > > 1. Alice -- in Vat A -- holds a capability C which is a promise pointing > towards Vat B. > 2. Alice calls foo() on C. This call is sent to Vat B. > 3. Vat B informs Vat A that C has resolved, and it points to Carol, who > also lives in Vat A. Thus, future calls can be made directly in-process. > (However, the call to foo() is still in-flight.) > 4. Alice calls bar() on C. Since the promise has resolved, this call can > be delivered locally directly to Carol. > 5. Vat B reflects the call to foo() back to Carol in Vat A. > > In a naive implementation, the bar() call will arrive at Carol before the > foo() call does, which is wrong. > > We need to introduce embargoes to prevent this: > > 1. Alice -- in Vat A -- holds a capability C which is a promise pointing > towards Vat B. > 2. Alice calls foo() on C. This call is sent to Vat B. > 3. Vat B informs Vat A that C has resolved, and it points to Carol, who > also lives in Vat A. Thus, future calls can be made directly in-process. > (However, the call to foo() is still in-flight.) > 3.1 Vat A marks C as pointing to Carol, but embargoed. > 3.2 Vat A sends a Disembargo message towards Vat B, addressed to the > original promise. (It then releases the promise.) > 4. Alice calls bar() on C. Since the promise has resolved, this call can > be delivered locally directly to Carol. > 4.1 Because C is marked embargoed, the message is held in a queue, not > delivered yet. > 5. Vat B reflects the call to foo() back to Carol in Vat A. > 5.1 Vat A delivers foo() to Carol. > 5.2 Vat B reflects the Disembargo back to Carol in Vat A. > 5.3 Vat A processes the Disembargo, which releases the embargo on the > capability C. > 5.4 The bar() call, which was previously embargoed, is now delivered to > Carol. > > >> >> > There are two particular cases where embargos are important. Consider >>> object Alice, in Vat A, >>> > who holds a promise P, pointing towards Vat B, that eventually >>> resolves to Carol. >>> >>> Could Carol be another promise here? Should Alice wait until the target >>> is fully resolved before doing a disembargo, or do a disembargo for each >>> step? >>> >> >> See above explanation. But no, Carol cannot be a promise, since the only >> time that an embargo is triggered is once you get back a locally hosted >> capability. >> > > Actually Carol could be a promise -- a promise currently hosted in Vat A. > (The original promise, which resolved, was hosted in Vat B, but it resolved > to another promise, back in Vat A.) > > This situation leads to the Tribble 4-way Race Condition, which is > described in the Disembargo doc comment. > > The short version is that once Vat B declares to Vat A that the promise > has resolved to Carol, then Vat B must always forward all messages > addressed to the promise directly to Carol. Even if Vat A later declares > that Carol has further resolved to Dave, Vat B cannot start forwarding > messages directly to Dave -- it must keep sending them to Carol. > > >> >> >>> [Accept] >>> > This message is also used to pick up a redirected return -- see >>> `Return.redirect`. >>> >>> `redirect` doesn't appear anyway else in this spec. I guess it's >>> `Return.sendResultsTo.thirdParty`. >>> >> >> Probably. It's Level 3, so it's invisible to me. :D >> > > Correct. > > >> >> >>> >>> [ Network-specific Parameters] >>> > For interaction over the global internet between parties with no other >>> prior arrangement, a >>> > particular set of bindings for these types is defined elsewhere. >>> (TODO(someday): Specify where >>> > these common definitions live.) >>> >>> Do these definitions exist now? >>> >> >> ¯\_(ツ)_/¯ >> > > They do not. :( > > I agree the docs should be updated here. It'd be great if someone who is > not me wanted to make the changes since other people can probably > better-identify which bits are confusing than I can... :) > > -Kenton > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto. -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
