[capnproto] [ACTION REQUIRED] Cap'n Proto mailing list moving to GitHub Discussions

['Kenton Varda' via Cap'n Proto]

Hi capnproto mailing list,

As you have likely noticed, the Cap'n Proto mailing list has had a spam
problem lately. I don't know what changed. We never had a problem for 10+
years, and then suddenly, Google couldn't keep the spammers out anymore.

Seeing as how Google Groups appears to be a dead product kept around only
because Google uses it internally, it's probably time to give up on it. To
that end, going forward we will use GitHub discussions instead:

https://github.com/capnproto/capnproto/discussions

Please update your notification settings accordingly.

-Kenton

(Gory details: The setting I've always used in Google Groups is to require
moderation for messages from non-members of the group. But if you *joined*
the group, you could post right away. I guess the spam bots have suddenly
learned how to join the group before posting. I could change things so all
new members start out moderated. But in my experience, people are
incredibly confused by moderation, as Google Groups doesn't seem to give
them any indication of why their message didn't post. Plus, I'm often very
slow to approve messages. Increasing moderation seems like a loss.)

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnj%3DMCd8Gna-LABaXoPpoHSDOKS_gzikjwhY59HeGJSyw%40mail.gmail.com.

Re: [capnproto] Trying to understand how to use CapnProto

['Kenton Varda' via Cap'n Proto]

You basically understand correctly, although what you describe is a little
bit closer to the Protobuf model than the Cap'n Proto model. They are very
similar, but the difference with capnp is that the generated classes do not
create a self-contained copy of the message content, but rather act as
cursors into the underlying message buffer. The setter methods write
directly into the underlying buffer, and the getter methods read directly
from it (or return pointers pointing into it). This differs from Protobuf
where the buffer is parsed into an object tree upfront, and the generated
class represents that object tree -- once you have that object tree, the
byte buffer can be discarded.

The reason this matters is, if you use a design where you convert
CnP_Config into your traditional Config class upfront, you lose this
zero-copy benefit. At that point in time, you are presumably making copies
of all the message contents into an in-memory object tree. If you can avoid
that, you will get a performance benefit. One way to avoid it would be to
use CnP_Config::Reader directly, replacing your old Config class. Another
approach might be to refactor the Config class so that it can be backed by
a CnP_Config::Reader under the hood, to avoid the upfront copy.

That said, even with the upfront copy, you should see a performance benefit
vs. YAML, which is much more expensive to parse.

-Kenton

On Thu, Nov 9, 2023 at 6:28 AM Software Developer 
wrote:

> Hi,
>
> I'm pretty new to (de-)serialisation; at least the kind that revolves
> around frameworks
> like CnP, Protobuf, Msgpack, etc. I'm quite familiar with the concept,
> especially if hand-rolling
> your own binary (de-)serialisation using something like TLVs/XDR as a
> format.
>
> I have an application in C++ that loads in a YAML config, performs some
> validation and pre-processing,
> then parses N config items to set up the state for a running daemon loop.
> When N=1 this can cause the program to take minutes in startup time.
>
> It has been suggested that we look into binary (de-)serialisation where we
> first serialise the Config object state
> (root object of all compiled configuration) and its transitive
> dependencies after the YAML config
> has been parsed/processed into a file, and then in future invocations of
> the program load this file
> so we can skip the whole YAML dance and deserialse the Config object, and
> go directly to the daemon loop.
>
> My understanding of schema based (de-)serialisation frameworks is that you
> write a schema to indicate what
> will be the format on the wire, and then using that schema you generate
> classes/structs that will represent
> that schema.
>
> How are these generated classes meant to be used exactly? Can I retrofit
> my existing Config class with CnP
> and just supply a (de-)serialise() function. Going by the usage examples
> on CnP website this doesn't seem to
> be the case.
>
> So this is my current thinking: if CnP generates a class called CnP_Config
> (distinct from the concrete Config class)
> which represents the schema/wire format, then these will be use as
> intermediate classes for (de-)serialisation.
> Specifically if I want to serialise the Config class, I would write a
> Config::serialise() function that returns a CnP_Config
> which will construct a CnP_Config object that is completely populated.
> Then with the CnP_Config object I would then
> write this out to the filesystem using the CnP utility functions.
> Similarly to deserialise, I would load the file on the wire
> into a CnP_Config object using the CnP utility functions, and then have a
> function like
> Config::deserialise(CnP_Config ) which will convert the CnP_Config
> object into a concrete Config object.
>
> Is this the right approach? Or should I be using these CnP generated
> classes as first-class classes and replace the
> Config class in my program with CnP generated ones? The problem with doing
> this is that these objects or sub-objects
> may have inheritence and is already quite entangled in the existing code
> base; supplanting them with CnP generated
> classes would be a herculean effort.
>
> Thanks,
> Software Developer
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/7085ea4d-5bad-482c-85e2-d152d4290386%40app.fastmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn1OxRD_ev-1UW_5Vx42SNg_SqaGinVFH0r1TrkByKbBQ%40mail.gmail.com.

Re: [capnproto] v2 modules and versioning

['Kenton Varda' via Cap'n Proto]

[Sorry for the long delay in replying -- I recently moved into a new house
and have been rather swamped.]

Cap'n Proto follows in the Protobuf philosophy of versioning, which is,
there are no versions, or alternatively, versions are a continuous
spectrum. As long as each incremental change is made in a
backwards-compatible way, then old programs should be able to talk to new
programs and vice versa. If you want to make a breaking change, you make a
whole new type to represent the new protocol. If you want to put "v2" or
whatever in the name of that type, that's up to you.

Cap'n Proto uses 64-bit type IDs to canonically identify a type. Two type
definitions are presumed to be versions of the same type if they have the
same ID. Hence, type *names* are merely a convenience for developers
writing code, but can freely be changed without breaking compatibility. In
pure Cap'n Proto, type IDs are the only global namespace of types, and
collisions there are unlikely due to being chosen randomly. In most
programming languages we are forced to place type names into some sort of
global namespace, but that's up to the language-specific code generator to
deal with.

> backwards compatibility of protocol versions can be mechanically
validated.

With Cap'n Proto's SchemaLoader (in C++, at least), when you load two types
with the same ID, it will automatically check them for compatibility and
choose the newer version as the type that the SchemaLoader ultimately gives
to the application. This is done by actually comparing the schema contents.
If the two versions are inherently incompatible, an exception is thrown. I
don't know if version numbers would actually add anything here.

Of course, it's entirely possible that a newer version of some software has
changed the interpretation of a schema, without changing the actual
definition in any way that is detectably incompatible. Obviously, it's
fundamentally impossible to detect such incompatibility in an automated way.

-Kenton

On Wed, Sep 20, 2023 at 7:50 AM Jonathan Shapiro 
wrote:

> I've been thinking about modules and versioning. CapnProto has an import
> mechanism, but it doesn't seem to have a first-class concept of a schema
> that can be versioned.
>
> Recently, I've spent a bunch of time working in both TypeScript and Go,
> and I had designed a module system for BitC many years ago with some care
> for verification. It took some getting used to, but from a developer
> perspective I have come to feel that Go has made a good set of
> pragmatic decisions by tying code repositories, the cryptographic hashes
> they supply, and version tags to modules and versioning, and by
> *separating* imported identifier aliasing from the import itself. The
> go.mod/go.sum combination seems to handle everything that the node package
> system does with regard to version binding, but subjectively feels simpler.
> I *do* find that import paths get long, and I sometimes wish that go.mod
> had a way to do import path shorthands, but I haven't ever hit a point
> where that seemed critical.
>
> [ I definitely do *not* like Go's decision to conflate identifier
> capitalization with export. It's cute in a bad way, irritating, and breaks
> link compatibility with *everything*. In schemas, the need to distinguish
> between public and private things doesn't arise in the same way, because
> the whole point is to be publishing the protocol. ]
>
>
> For CapnProto, I imagine we would call the versioned thing a schema rather
> than a module. A pleasant side benefit is that well-defined versions on
> schemas offer the possibility that the backwards compatibility of protocol
> versions (e.g.  v1.1.0 relative to v1.0.0) can be mechanically *validated* -
> which seems useful.
>
>
> Two relevant points are made in the CapnProto language description.
> Paraphrasing:
>
>1. "Symbolic names can collide... which can be hard to detect in large
>systems using different versions of protocols."
>
>This point is made in the context of discussing type IDs. CapnProto
>needs type IDs for wire encoding reasons, but *this* isn't the right
>argument for having them! It's an argument for a proper module and version
>system. And as an aside, the question of type equivalence in the presence
>of federated protocols is good for a couple of doctoral dissertations.
>2. "Fully qualified names become large and waste space on the wire."
>
>As has been noted elsewhere, CapnProto's "everything is a namespace"
>leads to *horrifically* long names produced by the generators, so I
>think that ship has already sailed. The Go module system and import design
>limits the length of names in code to "importBinding.typeName". It would
>also help to get rid of the "everything is a namespace" idea.
>
>The notion of wasted space on the wire because of long names seems
>like a red herring, because I can't see anything in the spec suggesting
>that identifier names ever *appear* on the 

Re: [capnproto] Exiting from EzRpcServer

['Kenton Varda' via Cap'n Proto]

Hi Richard,

In the RPC server sample code, there is a line like this:

kj::NEVER_DONE.wait(waitScope);

This waits forever, running the event loop.

If you don't want the loop to run forever, the trick is to wait on a
promise that resolves at some point. For instance, you could create one:

auto [promise, fulfiller] = kj::newPromiseAndCrossThreadFulfiller();
promise.wait(waitScope);

Now, some other thread can call `fulfiller->fulfill()` to cause the
`promise.wait()` to return.

It would also be possible to make the KJ event loop run in the same thread
as ASIO, but this would indeed be a lot of work -- you would essentially
have to implement the KJ I/O interfaces in terms of ASIO, instead of using
UnixEventPort.

-Kenton

On Sun, Oct 29, 2023 at 3:29 PM Richard Hacker  wrote:

> Hello
>
> while EzRpcServer is a very easy two liner to use on its own (see
> ez-rpc.h), it is daunting to use RPC outside this context.
>
> My application is a data collection agent. While it is gathering data
> from its sources as a TCP client, at the same time it also serves this
> data to other clients connecting to it using capnp RPC. It uses ASIO as
> an event loop, but the problem would not change when using kj's event
> loop. To detach the data collector from the RPC server, I want to launch
> the RPC server in a thread of its own.
>
> The problem I am facing is how to shut down the RPC server thread from
> listening and its (long running) clients explicitly so that the main
> thread can join() the RPC thread. While EzRpcServer implementation is
> very short, I have no clue as to where I should start.
>
> Since the RPC server is running in a thread of its own, it can run its
> own EventPort, although merging it into ASIO's event loop would be the
> silver bullet, but I guess quite an integrate task.
>
> In the long run, I would also like to use RPC Client from a Qt
> application, this time it should be running instde Qt's event loop, but
> that is another story.
>
> Any hints would be very welcome
>
> Mit freundlichem Gruß
>
> Richard Hacker
>
> --
> 
>
> Richard Hacker M.Sc.
> richard.hac...@igh.de
> Tel.: +49 201 / 36014-16
>
> Ingenieurgemeinschaft IgH
> Gesellschaft für Ingenieurleistungen mbH
> Nordsternstraße 66
> D-45329 Essen
>
> Amtsgericht Essen HRB 11500
> USt-Id.-Nr.: DE 174 626 722
> Geschäftsführung:
> - Dr.-Ing. Siegfried Rotthäuser
> - Dr. Sven Beermann, Prokurist
> Tel.: +49 201 / 360-14-0
> http://www.igh.de
>
> 
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/c67b1261-d39c-4138-8feb-55f045236986%40igh.de
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQndG%3DHQJLGgBA58fexnbA1BWCGsVyaQg0kKbBcE-OAWyQ%40mail.gmail.com.

Re: [capnproto] Usage of TwoPartyClient() for client callback mechanism

['Kenton Varda' via Cap'n Proto]

This is expected. The KJ event loop runs when you call `.wait()`. The event
loop checks for network events and delivers callbacks. If you haven't
called .wait(), then the event loop has no way to run. KJ does not create
any background threads for this purpose; the thread that created the event
loop must call .wait() on a promise in order to run it. Usually, large
programs will actually have a single top-level `.wait()` call that runs the
entire program -- this might literally be `kj::NEVER_DONE.wait(waitScope)`
to wait forever. In such programs, all actual work is done in `.then()`
callbacks.

If you need to be able to receive KJ events at the same time as executing
other code (especially bulk CPU-heavy work), then you will have to create a
separate thread yourself, and figure out how to communicate between them.
The KJ event loop is tied to a specific thread and cannot be directly
accessed from other threads; however, you can use the `kj::Executor`
interface to schedule work on another thread's event loop, or
kj::newPromiseAndCrossThreadFulfiller() to create a promise/fulfiller pair
that can be fulfilled from a different thread.

-Kenton

On Tue, Oct 17, 2023 at 6:37 AM Bhavya R  wrote:

> Hello All,
>
> I am trying to pass client capability to sever, client should receive
> notify back  on any updates.
> In my application, Service is sending statusHearbeat continuously, but
> client receives callback only when  it is waiting on any promise. Could
> anyone please guide on how to receives callback from server? My client
> application is running on windows.
>
> For e.g  only duration when i am blocked on wait() call, i see callback
> gets received and dispatchCall is triggered
>
> This call is done from client -> server
> auto request = connect->directorRef->versionRequest();
> auto responsePromise = request.send();
>auto response = *responsePromise.wait(*
> connect->clientRef->getWaitScope());
>
> e.g interface FaTfServerToClient {
>  statusHeartbeat   @0 () -> ();
> }
>
> interface FaTfClientToServer {
>  version @0 () -> (version :Base.Version);
> }
>
> Client connection---> I am passing client bootstrap TransformEventHandler
> object
>
>
>  auto address = connect->ioContext.provider->getNetwork()
> .parseAddress(ipAddress,
> port).wait(connect->ioContext.waitScope);
> connect->connection =
> address->connect().wait(connect->ioContext.waitScope);
> connect->clientRef =
> std::make_shared(*(connect->connection), kj::heap<
> *TransformEventHandler*>(observer),
> capnp::schemas::Side_9fd69ebc87b9719c::CLIENT);
> connect->directorRef =
> std::make_shared(connect->clientRef->bootstrap().castAs<
> FaTfClientToServer >());
>
>
> *class TransformEventHandler : public FaTfServerToClient::Server, public
> EventHandler {*
> ::kj::Promise statusHeartbeat(StatusHeartbeatContext context)
> override {
> std::cout << "status heartbeat" << std::endl;
> return kj::READY_NOW;
> }
>  }
>
> I tried adding another iocontext, client in different thread and ran
> wait() on that. It doesnt receive nothing.
> Looks like i haven't understood basic mechanism!! Any pointers would be
> very helpful. Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/97c4321a-fcf7-4904-a9fc-11ece2bc6aecn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnzCdEN7QxEVu9XDr6mYCLoBoSDooa0FdYy0vRaBShCGw%40mail.gmail.com.

Re: [capnproto] Clean Disconnect Procedure

['Kenton Varda' via Cap'n Proto]

On Fri, Sep 29, 2023 at 3:36 PM Alex  wrote:

> A. How shall a peer discern between an exceptional connection closure
> and a normal one?
>
> My proposal is to add a new message, "GOODBYE", which allows the system
> to signal to its remote peer an intent to close the connection.


Right, I get the motivation, but what I'm saying is there is no behavior or
API in Cap'n Proto which cares to distinguish between these. Are you
proposing adding a new API which would allow the application to learn
whether the disconnect was intentional?

I think I would prefer that we leave this up to the application instead.
That is, an application can always have an RPC method which signals a clean
end, without any special support from the underlying transport. This
approach keeps the application decoupled from the transport layer, allowing
the same code to work with local capabilities or alternative protocols.

Note
> that this message is communicated within the context of any
> encryption/authentication layer which may be in place (such as TLS),
> thus allowing application designers to discern between proper operation
> and outside interference.
>

TLS itself already communicates the difference between a clean shutdown and
an outside interruption. In KJ, read()ing from a TLS socket will throw an
exception if it is prematurely terminated.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQk1nkTFq_Q5%3DtNdkQPa_6Vco0E0GcLfvb%3DY-NbLL-03PA%40mail.gmail.com.

Re: [capnproto] Clean Disconnect Procedure

['Kenton Varda' via Cap'n Proto]

erride {
> > >   this->shutdown->fulfill();
> > > }
> > >
> > >   private:
> > > kj::Own> shutdown;
> > > };
> > >
> > > Another approach could be to add a shutdown() method to FooContext.
> > >
> > > On the client side, perhaps it is best to simply allow the
> > > rpcSystem to fall out of scope, at which point the destructors can
> > > invoke the necessary machinery to send the GOODBYE and FIN the TCP
> > > stream.
> > >
> > > What do you think? Are these approaches going to lead to a leaky
> > > abstraction? Do you know of an elegant way to design this?
> > >
> > > Regards,
> > > Alex
> > >
> > > On Wed, 27 Sep 2023 14:07:07 -0500
> > > "'Kenton Varda' via Cap'n Proto"  wrote:
> > >
> > > > (Happy to accept a PR. The relevant code is in `messageLoop()` and
> > > > `RpcConnectionState::disconnect()` in `rpc.c++`.)
> > > >
> > > > On Wed, Sep 27, 2023 at 2:05 PM Kenton Varda
> > > >  wrote:
> > > >
> > > > > Indeed, there isn't really a clean shutdown mechanism right
> > > > > now. I guess it hasn't come up as a priority because in most
> > > > > use cases we just haven't really cared if there's a TCP RST
> > > > > triggered under the hood... since we're already killing the
> > > > > connection, we ignore that error anyway.
> > > > >
> > > > > I suppose what we should do is, in the case that we receive a
> > > > > clean EOF, inhibit the sending of an abort message back, just
> > > > > send EOF back.
> > > > >
> > > > > -Kenton
> > > > >
> > > > > On Thu, Sep 21, 2023 at 6:09 PM 'Alex' via Cap'n Proto <
> > > > > capnproto@googlegroups.com> wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> I am designing an application (in C++) where, upon invocation
> > > > >> of a particular RPC call, both the server and the client agree
> > > > >> to cleanly disconnect from one another. By "cleanly", I mean
> > > > >> that both the server and the client send a TCP FIN/ACK and
> > > > >> nothing more (e.g. no RSTs). Unfortunately, in the current
> > > > >> design the receipt of a FIN will cause
> > > > >> AsyncIoMessageStream::tryReadMessage() to abort, whereupon it
> > > > >> will throw KJ_EXCEPTION(DISCONNECTED, "Peer
> > > > >> disconnected.")[0]. This exception is eventually written to
> > > > >> the client socket, and if the client is already gone, there
> > > > >> will be one or more RSTs in response:
> > > > >>
> > > > >> C -> S: "Goodbye" (RPC call)
> > > > >> C -> S: "I have nothing more to say" (TCP FIN)
> > > > >>
> > > > >> (the client does not expect the server to say anything more and
> > > > >> closes the socket)
> > > > >>
> > > > >> S -> C: "Exception! You disconnected from me" (RPC message)
> > > > >> C -> S: "Error: Connection reset by peer" (TCP RST)
> > > > >>
> > > > >> Given that both the server and client have agreed to shut down
> > > > >> the connection, this is not an exceptional circumstance.
> > > > >> Therefore, an exception should not be thrown.
> > > > >>
> > > > >> Unfortunately, there does not seem to be a way to indicate to
> > > > >> the RpcSystem that the DISCONNECTED exception ought to be
> > > > >> suppressed. Is there something I am missing? I appreciate any
> > > > >> assistance.
> > > > >>
> > > > >> Regards,
> > > > >> Alex
> > > > >>
> > > > >> [0]
> > > > >>
> > >
> https://github.com/capnproto/capnproto/blob/761aeb17563a59f43b3fe9bae93df83c6bd57d06/c%2B%2B/src/capnp/rpc.c%2B%2B#L2775
> > >
> > > > >>
> > > > >> --
> > > > >> You received this message because you are subscribed to the
> > > > >> Google Groups "Cap'n Proto" group.
> > > > >> To unsubscribe from this group and stop receiving emails from
> > > > >> it, send an email to capnproto+unsubscr...@googlegroups.com.
> > > > >> To view this discussion on the web visit
> > > > >>
> > >
> https://groups.google.com/d/msgid/capnproto/20230921190853.115b911d%40centromere.net
> > >
> > > > >> .
> > > > >>
> > > > >
> > > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "Cap'n Proto" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> > > send an email to capnproto+unsubscr...@googlegroups.com.
> > > To view this discussion on the web visit
> > >
> https://groups.google.com/d/msgid/capnproto/20230927173552.7ad4b38e%40centromere.net
> > > .
> > >
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3Du-8kq-9jahcgPw58Xa1BQy_paB%2Bzk9frUB8157Ap7Eg%40mail.gmail.com.

Re: [capnproto] Clean Disconnect Procedure

['Kenton Varda' via Cap'n Proto]

On Wed, Sep 27, 2023 at 4:37 PM 'Alex' via Cap'n Proto <
capnproto@googlegroups.com> wrote:

> 1. I would like to add a new RPC Message in rpc.capnp:
>
> goodbye @14 :Void;
>
> This message indicates to the recipient that the sender has nothing
> more to say, and that it should stop read()ing the socket. In other
> words, upon receipt of rpc::Message::GOODBYE, messageLoop() ends
> gracefully (no exceptions are thrown). The sender then shuts down its
> write() side of the connection, causing a TCP FIN to be delivered to
> the recipient. Because there is no read() in progress, an exception shall
> not be thrown.
>
> The recipient performs whatever cleanup is necessary and sends a
> reciprocal GOODBYE, causing the same logic described above to be
> invoked on the other end.
>
> Do you think this is a good solution?
>

Hmm, what's the benefit of this, vs. simply sending EOF?

To extend the protocol in this way we would have to think about backwards
compatibility. If a peer running an older version of capnp receives the
"goodbye" message, it will respond with an "unimplemented" message, which
seems like it could make things worse?


> 2. In your opinion, what is the best way to expose this graceful
> disconnect functionality to applications?


This is a bit tricky.

For rpc-twoparty.h I think it's straightforward, it could simply be a
method on `TwoPartyClient` and `TwoPartyServer` to signal graceful
disconnect. (This would have to be a method returning a promise which
resolves when all buffers are flushed and such, so I don't think it can
just be destructor behavior.)

But in the full many-party vision of Cap'n Proto, the application is not
really intended to know what connections exist. The application could
receive two capabilities from two different parties which both happened to
point to the same third party, and those two capabilities end up sharing a
connection, even though they came from different places. So it seems like
the application has no reasonable way to express that it wants a connection
to shut down, if it doesn't even know a connection exists.

I think, then, it has to be up to the RPC system to shut down connections
that are idle. Probably RpcSystem could signal to the underlying VatNetwork
whenever a connection has reached an idle state, meaning it has no
outstanding RPCs nor capabilities. The VatNetwork could choose to close
such a connection if it feels like it -- some transports may want to do
this on a timeout, others may decide it's better to keep the connection
open.

But I'd suggest not worrying about that for now and focusing just on
rpc-twoparty, since that's what most people are using today.

-Kenton


> I considered modifying the
> signature of the BootstrapFactory's createFor method in this manner:
>
> Before:
>
> capnp::Capability::Client createFor(VatId::Reader clientId)
>
> After:
>
> capnp::Capability::Client createFor(VatId::Reader clientId,
> kj::Own> shutdown)
>
> The PromiseFulfiller can then be passed to the constructor of the Server:
>
> class AdderImpl final: public Adder::Server {
>   public:
> AdderImpl(kj::Own> shutdown) :
> shutdown(kj::mv(shutdown)) {}
>
> kj::Promise add(AddContext context) override {
>   auto params = context.getParams();
>   context.getResults().setValue(params.getLeft() + params.getRight());
>   return kj::READY_NOW;
> }
>
> kj::Promise cleanupGracefully(CleanupGracefullyContext context)
> override {
>   this->shutdown->fulfill();
> }
>
>   private:
> kj::Own> shutdown;
> };
>
> Another approach could be to add a shutdown() method to FooContext.
>
> On the client side, perhaps it is best to simply allow the rpcSystem to
> fall out of scope, at which point the destructors can invoke the
> necessary machinery to send the GOODBYE and FIN the TCP stream.
>
> What do you think? Are these approaches going to lead to a leaky
> abstraction? Do you know of an elegant way to design this?
>
> Regards,
> Alex
>
> On Wed, 27 Sep 2023 14:07:07 -0500
> "'Kenton Varda' via Cap'n Proto"  wrote:
>
> > (Happy to accept a PR. The relevant code is in `messageLoop()` and
> > `RpcConnectionState::disconnect()` in `rpc.c++`.)
> >
> > On Wed, Sep 27, 2023 at 2:05 PM Kenton Varda 
> > wrote:
> >
> > > Indeed, there isn't really a clean shutdown mechanism right now. I
> > > guess it hasn't come up as a priority because in most use cases we
> > > just haven't really cared if there's a TCP RST triggered under the
> > > hood... since we're already killing the connection, we ignore that
> > > error anyway.
> > >
> > > I suppose what we should do is, in the case that we receiv

Re: [capnproto] Clean Disconnect Procedure

['Kenton Varda' via Cap'n Proto]

(Happy to accept a PR. The relevant code is in `messageLoop()` and
`RpcConnectionState::disconnect()` in `rpc.c++`.)

On Wed, Sep 27, 2023 at 2:05 PM Kenton Varda  wrote:

> Indeed, there isn't really a clean shutdown mechanism right now. I guess
> it hasn't come up as a priority because in most use cases we just haven't
> really cared if there's a TCP RST triggered under the hood... since we're
> already killing the connection, we ignore that error anyway.
>
> I suppose what we should do is, in the case that we receive a clean EOF,
> inhibit the sending of an abort message back, just send EOF back.
>
> -Kenton
>
> On Thu, Sep 21, 2023 at 6:09 PM 'Alex' via Cap'n Proto <
> capnproto@googlegroups.com> wrote:
>
>> Hi all,
>>
>> I am designing an application (in C++) where, upon invocation of a
>> particular RPC call, both the server and the client agree to cleanly
>> disconnect from one another. By "cleanly", I mean that both the server
>> and the client send a TCP FIN/ACK and nothing more (e.g. no RSTs).
>> Unfortunately, in the current design the receipt of a FIN will cause
>> AsyncIoMessageStream::tryReadMessage() to abort, whereupon it will
>> throw KJ_EXCEPTION(DISCONNECTED, "Peer disconnected.")[0]. This
>> exception is eventually written to the client socket, and if the client
>> is already gone, there will be one or more RSTs in response:
>>
>> C -> S: "Goodbye" (RPC call)
>> C -> S: "I have nothing more to say" (TCP FIN)
>>
>> (the client does not expect the server to say anything more and closes
>> the socket)
>>
>> S -> C: "Exception! You disconnected from me" (RPC message)
>> C -> S: "Error: Connection reset by peer" (TCP RST)
>>
>> Given that both the server and client have agreed to shut down the
>> connection, this is not an exceptional circumstance. Therefore, an
>> exception should not be thrown.
>>
>> Unfortunately, there does not seem to be a way to indicate to the
>> RpcSystem that the DISCONNECTED exception ought to be suppressed. Is
>> there something I am missing? I appreciate any assistance.
>>
>> Regards,
>> Alex
>>
>> [0]
>> https://github.com/capnproto/capnproto/blob/761aeb17563a59f43b3fe9bae93df83c6bd57d06/c%2B%2B/src/capnp/rpc.c%2B%2B#L2775
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Cap'n Proto" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to capnproto+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/capnproto/20230921190853.115b911d%40centromere.net
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkvmFfnXVLR6M8itZeog4KrXLFXmaFiQw8ZxSxOfOyFMw%40mail.gmail.com.

Re: [capnproto] Clean Disconnect Procedure

['Kenton Varda' via Cap'n Proto]

Indeed, there isn't really a clean shutdown mechanism right now. I guess it
hasn't come up as a priority because in most use cases we just haven't
really cared if there's a TCP RST triggered under the hood... since we're
already killing the connection, we ignore that error anyway.

I suppose what we should do is, in the case that we receive a clean EOF,
inhibit the sending of an abort message back, just send EOF back.

-Kenton

On Thu, Sep 21, 2023 at 6:09 PM 'Alex' via Cap'n Proto <
capnproto@googlegroups.com> wrote:

> Hi all,
>
> I am designing an application (in C++) where, upon invocation of a
> particular RPC call, both the server and the client agree to cleanly
> disconnect from one another. By "cleanly", I mean that both the server
> and the client send a TCP FIN/ACK and nothing more (e.g. no RSTs).
> Unfortunately, in the current design the receipt of a FIN will cause
> AsyncIoMessageStream::tryReadMessage() to abort, whereupon it will
> throw KJ_EXCEPTION(DISCONNECTED, "Peer disconnected.")[0]. This
> exception is eventually written to the client socket, and if the client
> is already gone, there will be one or more RSTs in response:
>
> C -> S: "Goodbye" (RPC call)
> C -> S: "I have nothing more to say" (TCP FIN)
>
> (the client does not expect the server to say anything more and closes
> the socket)
>
> S -> C: "Exception! You disconnected from me" (RPC message)
> C -> S: "Error: Connection reset by peer" (TCP RST)
>
> Given that both the server and client have agreed to shut down the
> connection, this is not an exceptional circumstance. Therefore, an
> exception should not be thrown.
>
> Unfortunately, there does not seem to be a way to indicate to the
> RpcSystem that the DISCONNECTED exception ought to be suppressed. Is
> there something I am missing? I appreciate any assistance.
>
> Regards,
> Alex
>
> [0]
> https://github.com/capnproto/capnproto/blob/761aeb17563a59f43b3fe9bae93df83c6bd57d06/c%2B%2B/src/capnp/rpc.c%2B%2B#L2775
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/20230921190853.115b911d%40centromere.net
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQncgKYxvr_LUj4F2dJJRjDLcS%2BPPLYLYKhPkvkF0XOp3g%40mail.gmail.com.

Re: [capnproto] Dynamic schemas

['Kenton Varda' via Cap'n Proto]

Hi Jonathan,

I suppose the first question is: Does your database actually need to
understand the dynamic fields at all? Or does it just need to be able to
echo them back to the client later on, with only the client actually
understanding them?

If only the client really needs to know about them, then I'd suggest using
an `AnyPointer` field which the client fills in however they want. (You
could also use generics for the same effect wrapped in nicer PL theory.)

If the server needs to understand the fields -- perhaps, to index them, or
something -- then one possibility is the client could upload its schema to
the server in capnp schema format (schema.capnp). On the server, you could
use the "dynamic API" (in C++, capnp/dynamic.h; some other implementations
support it as well but not all) to load this schema dynamically and
introspect the messages based on it.

Alternatively, you could of course just have the client send names and
values, as you suggest. Though at that point you might even consider just
using JSON (or one of the myriad binary alternative encodings of JSON) as
the Cap'n Proto encoding may actually be adding more bloat than it saves.

It's hard to say what's the best approach without really understanding the
details of the application, though.

-Kenton

On Sun, Sep 17, 2023 at 5:04 PM Jonathan Shapiro 
wrote:

> We're working on multiple applications with dynamic schemas: schemas where
> customers need to be able to add or delete fields to meet their particular
> requirements. This sort of thing impacts both the database layer and the
> wire schema. I can think of an implementation that doesn't suck (see
> below), I'm wondering what the best way would be to handle this sort of
> thing in capnproto.
>
> The doesn't suck approach:
>
> In this application we can restrict in advance what the valid types are
> for these new columns, which would allow us to handle the dynamic fields as
> a list struct/union values, each carrying its field name as a string or
> something similar. It's not an especially elegant way of describing the
> protocol, but it covers the use cases. It isn't *necessary* that the bits
> on the wire be "pretty", it's reasonably easy to decode into JavaScript
> objects on the client, and it doesn't seem like a huge lift on the server
> to validate that all expected extension fields are present and have values
> of the expected type.
>
> Is there a better way to think about this?
>
>
> Jonathan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/CAJdcQk3s9jnXR6VJEyhXSFHPO0qNNBBnRrm7A3Pr%2BukU5Q9DmQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkkAuc06H2ZzJuUPb8ir2M%2BzoAax0vXVuWUTOn2WdpsGg%40mail.gmail.com.

Re: [capnproto] Using Cap'n Proto for FFI

['Kenton Varda' via Cap'n Proto]

Hi Jonah,

Sorry for the very slow reply. This looks cool!

Thanks for taking over maintaining the C implementation -- I've added a
link to the web site.

-Kenton

On Fri, Aug 25, 2023 at 10:32 PM Jonah Beckford  wrote:

> I have an implementation of FFI (foreign function interface) that uses Cap
> n' Proto. From a Cap n' Proto perspective it is analogous to Cap n' Proto
> RPC but "solves" sharing in-process objects between different languages
> rather than distributed messaging. You can read the announcement at
> https://discuss.ocaml.org/t/ann-new-project-old-technique-dksdk-ffi-ocaml/12909
> (the OCaml programming language discussion boards) if you'd like.
>
> All the Cap n' Proto language bindings (ex. capn-ocaml) are important in
> the "DkSDK FFI" framework, but the most critical is the C language binding.
> Why? Plain old C is the best language to implement FFI that works across
> many languages. However, c-capnproto is unmaintained
>  (
> https://github.com/opensourcerouting/c-capnproto/issues/55). So I forked
> c-capnproto, made several fixes, added support for MSVC and improved the
> test/build matrix. That fork is available at
> https://gitlab.com/dkml/ext/c-capnproto. If no one has any objections it
> may be best to list that active fork as the C implementation on
> https://capnproto.org/otherlang.html.
>
> Thanks! Jonah
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/10a9d02a-1073-408c-acbb-7a30717f3400n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmmmHFYM%3DpwLAQRoiNVCVaYFjqasGt2MmdZqQfWktbDCg%40mail.gmail.com.

Re: [capnproto] Random access clarified

['Kenton Varda' via Cap'n Proto]

On Tue, Sep 5, 2023 at 6:08 AM Johannes Dröge  wrote:

> Great, thanks for the clear statement!
>
> I have some follow-up questions now:
>
> 1) How is the memory usage patter for the deserialization of such members
> when they are accessed (either from disk or in memory). I suppose that the
> accessed element must be converted to a hardware-specific representation in
> memory to be usable, right? I assume that such a copy will exist in memory
> when used but no other copies.
>

The Cap'n Proto wire encoding is documented on the web site:

https://capnproto.org/encoding.html

As you'll see, there is no need to translate to a "hardware-specific
representation", as the wire representation is already designed to be
agreeable to all modern hardware without translation. This is the core
design goal of Cap'n Proto serialization.

When reading a message in Cap'n Proto, the backing buffer is only accessed
on-demand when you call the getter method to get a field. There is no
preprocessing at all. When you call the getter method for a pointer field,
only the pointer itself is read, in order to construct a Reader object; the
destination data is not read until you call methods on the Reader object to
read it.


> 2) For embedded binary (aka Data) objects, I assume that no real
> deserialization is actually needed. Is there a way to read such objects in
> a stream-like fashion to avoid putting them into memory entirely?
>

When reading a byte array, you essentially get a pointer into the backing
buffer; none of the bytes are accessed until your code does the accessing.
If you are reading from an mmaped file, then the pages will only be loaded
into memory when you access them, and the kernel can automatically unload
pages later when it needs memory for something else. There's nothing
special you need to do to achieve "streaming" in this case.

If you are reading a message from the network, though, it is necessary for
the entire message to arrive in memory before you can begin accessing it.
To achieve "streaming" from the network, you need to design your
application to send chunks of the stream as separate RPC calls.


> 3) I assume that the python interface does work the same. Are you aware of
> any limitations?
>

The Python implementation wraps the C++ implementation, so should broadly
work the same, but not all APIs are exposed. I don't personally maintain
the Python code so I can't really answer detailed questions about what it
can or can't do, sorry.

-Kenton


>
> Thanks for your support, I really enjoy that piece of software und hope
> that I can also use it for RPC in the future!
> Kenton Varda schrieb am Montag, 4. September 2023 um 17:47:22 UTC+2:
>
>> Hi Johannes,
>>
>> Yes, it applies to list indexing.
>>
>> -Kenton
>>
>> On Mon, Sep 4, 2023 at 10:43 AM Johannes Dröge  wrote:
>>
>>> Hi there!
>>>
>>> The FAQ states *"*Random access*:* You can read just one field of a
>>> message without parsing the whole thing". *However, does that also
>>> apply to List indexing*? I have a flexible-length list of potentially
>>> large objects, and I need to access the nth list element from disk without
>>> having to hold other elements in memory.
>>>
>>> I started using capnp for internal serialization in a prototype, with a
>>> more dynamic approach to data types and data structures. For this, I'm
>>> mostly attracted by the fast implementation and random access option, which
>>> gives me the possibility to mmap data structures to lazy-load attributes
>>> from disk. I'm currently using the Python interface but I might switch to
>>> C++, Rust or go at a later stage.
>>>
>>> I will try to profile this with a toy example. Nevertheless, I'd be
>>> thankful for a theoretical consideration here!
>>>
>>> Cheers
>>> Johannes
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/48035174-9253-48d3-ad3d-b3fe69d249a3n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/4af62cf4-9d0d-4f42-a205-76d25d5520f4n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an 

Re: [capnproto] Random access clarified

['Kenton Varda' via Cap'n Proto]

Hi Johannes,

Yes, it applies to list indexing.

-Kenton

On Mon, Sep 4, 2023 at 10:43 AM Johannes Dröge  wrote:

> Hi there!
>
> The FAQ states *"*Random access*:* You can read just one field of a
> message without parsing the whole thing". *However, does that also apply
> to List indexing*? I have a flexible-length list of potentially large
> objects, and I need to access the nth list element from disk without having
> to hold other elements in memory.
>
> I started using capnp for internal serialization in a prototype, with a
> more dynamic approach to data types and data structures. For this, I'm
> mostly attracted by the fast implementation and random access option, which
> gives me the possibility to mmap data structures to lazy-load attributes
> from disk. I'm currently using the Python interface but I might switch to
> C++, Rust or go at a later stage.
>
> I will try to profile this with a toy example. Nevertheless, I'd be
> thankful for a theoretical consideration here!
>
> Cheers
> Johannes
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/48035174-9253-48d3-ad3d-b3fe69d249a3n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnJ%2BvDy64wBw-i9o0HaVapznMz2--CoooX5XQ0nhASEKA%40mail.gmail.com.

Re: [capnproto] Re: NAT-Traversal

['Kenton Varda' via Cap'n Proto]

When you run `capnp compile -orust`, the `capnp` tool will attempt to
execute a binary `capnpc-rust` from your $PATH. This binary seems to be
provided by the capnpc crate, but declaring a dependency on the crate
probably does not automatically install it into your $PATH. You can either
install the binary into $PATH, or you can provide the binary location
explicitly like:

capnp compile -o/path/to/capnpc-rust src/schema/user.capnp

I don't think there's a mailing list specifically for the Rust
implementation but you could try filing issues on the github repo.

-Kenton

On Thu, Aug 31, 2023 at 7:48 AM Tanguille Grootaert <
tanguille.groota...@gmail.com> wrote:

> That can't be the issue since my depedencies are containing the same ones
> as the examples in that repo, where `capnp` and `capnpc` are a part of. The
> code compiles so the dependencies are valid. Only thing I can think about
> is that the apt repo has outdated binaries for the Cap'n Proto compiler
> causing it to use the old naming.
>
> Is there a another place I could pose this question, maybe rust specific
> by any chance?
>
> Op donderdag 31 augustus 2023 om 14:30:42 UTC+2 schreef
> ken...@cloudflare.com:
>
>> You need to install the Rust capnp plugin. I'm not very familiar with
>> Rust but it looks like this is provided by the crate called `capnpc`.
>>
>> https://github.com/capnproto/capnproto-rust
>> https://docs.rs/capnpc/latest/capnpc/
>>
>> -Kenton
>>
>> On Thu, Aug 31, 2023 at 6:55 AM Tanguille Grootaert <
>> tanguille...@gmail.com> wrote:
>>
>>> Thanks for the explanation Kenton. Cool to see you're still so involved,
>>> it's an honor to get an answer from you.
>>>
>>> I am now trying to implement it in a little testproject with rust.
>>> However I can't seem to figure out how to install `capnpc-rust`. I
>>> installed both `capnproto` and `libcapnp-dev` via apt.
>>> If I do the command `capnp compile -orust src/schema/user.capnp` it
>>> outputs
>>> `rust: no such plugin (executable should be 'capnpc-rust')
>>> rust: plugin failed: exit code 1`
>>>
>>> This confuses me because other then a repo that has now been integrated
>>> in the main rust repo I can't find anything about this. What am I missing?
>>>
>>>
>>> Op woensdag 30 augustus 2023 om 16:43:09 UTC+2 schreef
>>> ken...@cloudflare.com:
>>>
 Hi Tanguille,

 Cap'n Proto is agnostic to the underlying byte transport. It can layer
 on top of any byte stream or datagram stream. In terms of the C++
 implementation, you can write a custom implementation of the
 capnp::MessageStream interface in terms of any transport you'd like. For
 example, you could use Cap'n Proto over WebRTC. It's up to you to establish
 the byte-oriented or datagram-oriented connection first, then Cap'n Proto
 helps you interact with the peer across that connection.

 NAT traversal is the responsibility of the underlying transport, so is
 outside the scope of Cap'n Proto itself.

 -Kenton

 On Wed, Aug 30, 2023 at 8:36 AM Tanguille Grootaert <
 tanguille...@gmail.com> wrote:

> I discovered that WebRTC is actually some kind of stream multiplexer.
> If I search for that in the context of Cap'n proto the only reference I 
> can
> find is in the release notes of the 0.8 release (
> https://capnproto.org/news/). There it is stated that it can't be
> considered stable yet, this is now 3 years ago but I can't find more
> information about this topic. Can anyone point me to a good source or an
> example implementation?
>
> Op woensdag 30 augustus 2023 om 12:27:54 UTC+2 schreef Tanguille
> Grootaert:
>
>> Hey all,ﾠ
>>
>> I wanna build a p2p network in Rust. After some research I discovered
>> I wanna work with libp2p, Cap'n Proto, ... In the docs of libp2p I read
>> that WebRTC has NAT-traversal built in which I need. Since I want to use
>> Cap'n Proto I don't wanna use this but use the RPC-protocol of Cap'n 
>> Proto.
>> Does that have support for NAT-traversal? I can't find anything about 
>> it.ﾠ
>>
>> If you have a better idea on how to approach this, that's ofcourse
>> always welcome. Thanks!
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to capnproto+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/88ce497f-d49a-4b97-a130-b787e6d9d0cen%40googlegroups.com
> 
> .
>
 --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to 

Re: [capnproto] Re: NAT-Traversal

['Kenton Varda' via Cap'n Proto]

You need to install the Rust capnp plugin. I'm not very familiar with Rust
but it looks like this is provided by the crate called `capnpc`.

https://github.com/capnproto/capnproto-rust
https://docs.rs/capnpc/latest/capnpc/

-Kenton

On Thu, Aug 31, 2023 at 6:55 AM Tanguille Grootaert <
tanguille.groota...@gmail.com> wrote:

> Thanks for the explanation Kenton. Cool to see you're still so involved,
> it's an honor to get an answer from you.
>
> I am now trying to implement it in a little testproject with rust. However
> I can't seem to figure out how to install `capnpc-rust`. I installed both
> `capnproto` and `libcapnp-dev` via apt.
> If I do the command `capnp compile -orust src/schema/user.capnp` it
> outputs
> `rust: no such plugin (executable should be 'capnpc-rust')
> rust: plugin failed: exit code 1`
>
> This confuses me because other then a repo that has now been integrated in
> the main rust repo I can't find anything about this. What am I missing?
>
>
> Op woensdag 30 augustus 2023 om 16:43:09 UTC+2 schreef
> ken...@cloudflare.com:
>
>> Hi Tanguille,
>>
>> Cap'n Proto is agnostic to the underlying byte transport. It can layer on
>> top of any byte stream or datagram stream. In terms of the C++
>> implementation, you can write a custom implementation of the
>> capnp::MessageStream interface in terms of any transport you'd like. For
>> example, you could use Cap'n Proto over WebRTC. It's up to you to establish
>> the byte-oriented or datagram-oriented connection first, then Cap'n Proto
>> helps you interact with the peer across that connection.
>>
>> NAT traversal is the responsibility of the underlying transport, so is
>> outside the scope of Cap'n Proto itself.
>>
>> -Kenton
>>
>> On Wed, Aug 30, 2023 at 8:36 AM Tanguille Grootaert <
>> tanguille...@gmail.com> wrote:
>>
>>> I discovered that WebRTC is actually some kind of stream multiplexer. If
>>> I search for that in the context of Cap'n proto the only reference I can
>>> find is in the release notes of the 0.8 release (
>>> https://capnproto.org/news/). There it is stated that it can't be
>>> considered stable yet, this is now 3 years ago but I can't find more
>>> information about this topic. Can anyone point me to a good source or an
>>> example implementation?
>>>
>>> Op woensdag 30 augustus 2023 om 12:27:54 UTC+2 schreef Tanguille
>>> Grootaert:
>>>
 Hey all,ﾠ

 I wanna build a p2p network in Rust. After some research I discovered I
 wanna work with libp2p, Cap'n Proto, ... In the docs of libp2p I read that
 WebRTC has NAT-traversal built in which I need. Since I want to use Cap'n
 Proto I don't wanna use this but use the RPC-protocol of Cap'n Proto. Does
 that have support for NAT-traversal? I can't find anything about it.ﾠ

 If you have a better idea on how to approach this, that's ofcourse
 always welcome. Thanks!

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/88ce497f-d49a-4b97-a130-b787e6d9d0cen%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/cd024e30-eb7d-4ca6-9c4d-9f7bf5a01da0n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmTo86ctjD%3DCe84mxtPAVTmK98vEqu6U6HWADs3Vfa%3D6A%40mail.gmail.com.

Re: [capnproto] Re: NAT-Traversal

['Kenton Varda' via Cap'n Proto]

Hi Tanguille,

Cap'n Proto is agnostic to the underlying byte transport. It can layer on
top of any byte stream or datagram stream. In terms of the C++
implementation, you can write a custom implementation of the
capnp::MessageStream interface in terms of any transport you'd like. For
example, you could use Cap'n Proto over WebRTC. It's up to you to establish
the byte-oriented or datagram-oriented connection first, then Cap'n Proto
helps you interact with the peer across that connection.

NAT traversal is the responsibility of the underlying transport, so is
outside the scope of Cap'n Proto itself.

-Kenton

On Wed, Aug 30, 2023 at 8:36 AM Tanguille Grootaert <
tanguille.groota...@gmail.com> wrote:

> I discovered that WebRTC is actually some kind of stream multiplexer. If I
> search for that in the context of Cap'n proto the only reference I can find
> is in the release notes of the 0.8 release (https://capnproto.org/news/).
> There it is stated that it can't be considered stable yet, this is now 3
> years ago but I can't find more information about this topic. Can anyone
> point me to a good source or an example implementation?
>
> Op woensdag 30 augustus 2023 om 12:27:54 UTC+2 schreef Tanguille Grootaert:
>
>> Hey all,
>>
>> I wanna build a p2p network in Rust. After some research I discovered I
>> wanna work with libp2p, Cap'n Proto, ... In the docs of libp2p I read that
>> WebRTC has NAT-traversal built in which I need. Since I want to use Cap'n
>> Proto I don't wanna use this but use the RPC-protocol of Cap'n Proto. Does
>> that have support for NAT-traversal? I can't find anything about it.
>>
>> If you have a better idea on how to approach this, that's ofcourse always
>> welcome. Thanks!
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/88ce497f-d49a-4b97-a130-b787e6d9d0cen%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnTAPXgQKTGnERnVp-F2OG-q%2BBRcDg1iCmCd-CUtUo7Uw%40mail.gmail.com.

Re: [capnproto] Schema Refactoring and Unique IDs

['Kenton Varda' via Cap'n Proto]

On Wed, Aug 2, 2023 at 11:51 PM Jonathan Shapiro 
wrote:

> Kenton: Are the type IDs likely to have collided?
>

If the two files contain types with identical names, they will have
identical IDs, as the default ID for a type is generated by taking a hash
of the parent scope's ID together with the type's name.


> Ian: If not, you might be able to hand-annotate the impacted types with
> the *already generated* IDs and then change the file ID. This offers the
> possibility of a clean changeover at the next breaking version change.
>
> Kenton: what *other* IDs will change if the file ID is changed that Ian
> might need to consider here?
>

64-bit high-entropy IDs are assigned to all "static" declarations, that is:
- files (top level)
- struct types
- enum types
- interface types
- constants
- annotations

Usually, in practice, only interface IDs really matter for compatibility
with other programs.

The other IDs might matter if programs at different versions of the schema
are dynamically exchanging their schemas. SchemaLoader decides that two
schemas are versions of the same type if they have the same type ID. But in
practice I've never seen a use case that simultaneously uses dynamic
schemas and is trying to reconcile multiple versions of those schemas.

-Kenton


>
>
> Jonathan
>
>
> On Wed, Aug 2, 2023 at 3:37 PM 'Kenton Varda' via Cap'n Proto <
> capnproto@googlegroups.com> wrote:
>
>> Yes, changing the file ID will change the IDs of all types declared
>> within, unless they have explicitly declared their own IDs.
>>
>> This only really matters if you are using RPC. Type IDs of interfaces are
>> part of the wire protocol. Type IDs of structs and enums are not really
>> used for anything, unless your application itself is using them. So it's
>> usually fine to change the type IDs of structs and enums.
>>
>> So if you are only using serialization, not RPC, just go ahead and change
>> one of the file IDs. If you are using RPC, you will need to manually
>> override the IDs of the interface types in the file to keep them consistent
>> with what they were originally, and then you can change the file ID.
>>
>> Of course, if both files happened to declare an interface with the same
>> name, those interfaces will have the same ID. In this case you have big
>> problems. You will have to make a breaking change to one of those two
>> interfaces.
>>
>> -Kenton
>>
>> On Tue, Aug 1, 2023 at 3:02 PM 'Ian Wilson' via Cap'n Proto <
>> capnproto@googlegroups.com> wrote:
>>
>>> Hi all, apologies for jumping on this thread a few years later.
>>>
>>> I have a similar question: suppose I have 2 different capnp files with
>>> the same file ID.
>>>
>>> # file foo.capnp
>>> 0xabbeabbeabbeabbe;
>>>
>>> struct Foo {
>>>   val @0 : UInt32;
>>> };
>>>
>>> # file bar.capnp
>>> 0xabbeabbeabbeabbe;
>>>
>>> struct Bar {
>>>   val @1 : UInt32;
>>> };
>>>
>>> In my project, someone copy/pasted a file leading to two different
>>> schemas with the same file ID, 0xabbeabbeabbeabbe.
>>> I find a use case where I want to import both of these capnp files and
>>> their types into a new one - of course this runs into a compilation error
>>> with the duplicate IDs.
>>>
>>> I'd like to generate a new file ID to replace one of these, but I'm not
>>> sure if this would impact the autogenerated type IDs and break other
>>> systems.
>>>
>>> Would merely changing the file ID affect the types that already exist in
>>> these files?
>>>
>>> On Friday, November 27, 2020 at 7:27:09 AM UTC-8 ken...@cloudflare.com
>>> wrote:
>>>
>>>> On Wed, Nov 25, 2020 at 3:43 PM Matt Stern  wrote:
>>>>
>>>>> When I try to run capnp compile, I get the following:
>>>>>
>>>>> error: Import failed: /capnp/java.capnp
>>>>>
>>>>
>>>> You will need to specify the same -I flags (import path) that you
>>>> normally specify to `capnp compile` when running the Java code generator.
>>>>
>>>> If I comment out the Java bits and just compile in C++ (which works for
>>>>> me), will this have any effect on the unique IDs for the structs in my
>>>>> schema file?
>>>>>
>>>>
>>>> No, the auto-generated IDs do not in any way depend on the contents of
>>>> other files. Auto-generated IDs are constructed by concatenating the parent
>>>> scope ID

Re: [capnproto] Schema Refactoring and Unique IDs

['Kenton Varda' via Cap'n Proto]

Yes, changing the file ID will change the IDs of all types declared within,
unless they have explicitly declared their own IDs.

This only really matters if you are using RPC. Type IDs of interfaces are
part of the wire protocol. Type IDs of structs and enums are not really
used for anything, unless your application itself is using them. So it's
usually fine to change the type IDs of structs and enums.

So if you are only using serialization, not RPC, just go ahead and change
one of the file IDs. If you are using RPC, you will need to manually
override the IDs of the interface types in the file to keep them consistent
with what they were originally, and then you can change the file ID.

Of course, if both files happened to declare an interface with the same
name, those interfaces will have the same ID. In this case you have big
problems. You will have to make a breaking change to one of those two
interfaces.

-Kenton

On Tue, Aug 1, 2023 at 3:02 PM 'Ian Wilson' via Cap'n Proto <
capnproto@googlegroups.com> wrote:

> Hi all, apologies for jumping on this thread a few years later.
>
> I have a similar question: suppose I have 2 different capnp files with the
> same file ID.
>
> # file foo.capnp
> 0xabbeabbeabbeabbe;
>
> struct Foo {
>   val @0 : UInt32;
> };
>
> # file bar.capnp
> 0xabbeabbeabbeabbe;
>
> struct Bar {
>   val @1 : UInt32;
> };
>
> In my project, someone copy/pasted a file leading to two different schemas
> with the same file ID, 0xabbeabbeabbeabbe.
> I find a use case where I want to import both of these capnp files and
> their types into a new one - of course this runs into a compilation error
> with the duplicate IDs.
>
> I'd like to generate a new file ID to replace one of these, but I'm not
> sure if this would impact the autogenerated type IDs and break other
> systems.
>
> Would merely changing the file ID affect the types that already exist in
> these files?
>
> On Friday, November 27, 2020 at 7:27:09 AM UTC-8 ken...@cloudflare.com
> wrote:
>
>> On Wed, Nov 25, 2020 at 3:43 PM Matt Stern  wrote:
>>
>>> When I try to run capnp compile, I get the following:
>>>
>>> error: Import failed: /capnp/java.capnp
>>>
>>
>> You will need to specify the same -I flags (import path) that you
>> normally specify to `capnp compile` when running the Java code generator.
>>
>> If I comment out the Java bits and just compile in C++ (which works for
>>> me), will this have any effect on the unique IDs for the structs in my
>>> schema file?
>>>
>>
>> No, the auto-generated IDs do not in any way depend on the contents of
>> other files. Auto-generated IDs are constructed by concatenating the parent
>> scope ID and the type name, and then taking a hash of that.
>>
>> -Kenton
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/2970b31f-a1aa-45ac-bc6b-a95beb0b02b9n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnk4s5N2adNTi8OwNG0SA1wzr-jrk8SzjVf3kQY%3DSrNGA%40mail.gmail.com.

[capnproto] Cap'n Proto 1.0 release candidate

['Kenton Varda' via Cap'n Proto]

Hello capnproto users,

I've created a release candidate for version 1.0 of Cap'n Proto's C++
implementation:

https://capnproto.org/capnproto-c++-1.0.0-rc1.tar.gz

There's nothing particularly special about this release! Frankly, I should
probably have applied the 1.0 designation to version 0.6 back in 2017, or
maybe even 0.5 back in 2014. It's now been over 10 years since the original
0.1 release and I guess I finally came to my senses, so I decided to go
ahead and make this one 1.0 instead of 0.11.

I'll have more details in the blog post when I do the final release next
week. In the meantime, please let me know if you find anything obviously
wrong with the RC.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmW29ouSUyaDt26DRgzWT13WuRgP0qUFAQBvbRXwSGaYQ%40mail.gmail.com.

Re: [capnproto] Flow Control Implementation

['Kenton Varda' via Cap'n Proto]

RpcSystem::setFlowLimit() should work for this. Yes, this means the RPC
system will stop reading messages from the socket buffer if too many
requests are in flight. That will naturally cause TCP backpressure to kick
in. However, buffering still could happen on the client side if the client
isn't careful to limit how much data is in flight.

Another, more advanced option is to declare your main payload methods as
streaming with `-> stream`. This causes the RPC system *on the client side*
to automatically keep track of how many calls are in flight and apply
backpressure if it's too high. More info here:
https://capnproto.org/news/2020-04-23-capnproto-0.8.html

You could also combine both approaches if you do not trust clients to apply
flow control correctly. Note that `-> stream` flow control is only actually
implemented in C++ so far, so clients in other languages might ignore it.

-Kenton

On Tue, Jul 11, 2023 at 10:18 AM Linus Meierhoefer <
linusmeierhoefe...@gmail.com> wrote:

> Hi,
> I am trying to setup an rpc server in order to integrate it into my custom
> publisher-subscriber library. For this I need a flow control mechanism that
> doesn't depend on individual streams but on a global disk writing speed.
> That is: The flow limit should adapt to the in-flight-bytes of the disk IO.
> Moreover, if the flow limit is exceeded the rpc server should effectively
> stop reading packets from client sockets, pushing the backpressure logic to
> TCP. This should prevent memory overhead for the broker when a publisher
> tries to flood the server.
> I saw that there is flow control logic implemented in capnp but wonder how
> it actually operates? The header-documentation of *RPCSystem *says that
> "the RpcSystem will not read further messages from the stream" but does
> this imply that these overflow messages will reside in the sockets buffer
> and don't get transferred to the receivers memory? And how is backpressure
> implemented then?
> If the default flow control logic does not fit my use case, I guess I have
> to setup a custom event loop that stops polling from sockets fds when the
> disk flow limit is exceeded.
> It would be really helpful if you could share some of your knowledge.
> Thanks in advance
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/7f0cd6f2-f5a6-4b30-a7ac-78ca1fc33af3n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D3T12Rn4jSnPrQYWabsAg1Je%3DRA8qjM-meAqcc9K9CXg%40mail.gmail.com.

Re: [capnproto] Common Lisp implementation

['Kenton Varda' via Cap'n Proto]

Cool! Let me know if you have any questions or anything doesn't make sense.

-Kenton

On Mon, Jul 10, 2023 at 11:02 AM Wojciech Gac 
wrote:

> Hi there,
>
> I'm about to attempt a Common Lisp implementation of Cap'n Proto. I was
> planning to start with serialization and then, provided it goes well, move
> on to RPC. My idea is to first gain a deeper understanding of the schema
> language by thoroughly reading the docs and then start hacking, probably
> looking at other folks' implementations along the way. I don't yet have a
> clear picture of the amount of work needed, but I wanted to make a sort of
> commitment to at least spend some time and effort on this project.
>
> Regards,
> Wojtek
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/9619e58b-c522-4d21-8f49-b7f6288be081n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3Df4BJ9%2BHVPFydsfFDHg2GNHtNiLwMEeo%2BzdiEmaHyoZg%40mail.gmail.com.

Re: [capnproto] Trying to learn how to use capnproto, but running into what seemed to be dead links

['Kenton Varda' via Cap'n Proto]

It sounds like you were reading the docs inside the github repo. The docs
directory there is used to generate the web site at https://capnproto.org .
If you read the web site, instead of the github repo, then all the links
should work correctly.

-Kenton

On Sun, Jun 25, 2023 at 7:57 AM CircuitCoder 4696 
wrote:

> I was trying to learn about capnproto.  Will you check your links please.
> I was trying to download capnproto, and couldn't download it from your
> website.  Before that, searching your GitHub, your link was to an
> `.html`-file, not a `.md`-file, and due to there being no `.html`-file
> about installing capnproto, GitHub gave me a 404 error.  Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/0afdb182-be13-43f9-80ae-37f01ec10264n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3Dwh05GzSctgaT4tKe0W6saLMMURY1ttHAztfcODA1qig%40mail.gmail.com.

Re: [capnproto] Memory Mapped Reading

['Kenton Varda' via Cap'n Proto]

Hi Adrian,

The memory usage you are seeing happens whether or not you use mmap, it's
just accounted differently. If you read the file using many small read()
calls, the operating system will still load all of the pages of the file
into memory, and will only remove them from memory when the memory is used
for something else. That's called caching. But when you use read(), the
memory isn't attached directly to your program, it's just in kernel space,
so it doesn't look like your program is using a lot of memory, even though
it is.

But using memory this way is not really consuming it. The memory is still
available for anything else that needs it. Since the memory is still
available, it's incorrect to think of it the same as memory your program
allocated for private use.

Put simply, your program is not using the memory you think it is. You need
to understand what the numbers actually mean.

-Kenton

On Wed, Jun 21, 2023 at 3:53 PM Adrian  wrote:

> Hi, thanks for your reply.
>
> I really appreciate your work in this library.
>
> I used /bin/time utility of Linux but I also saw the same result with
> another memory analyzer.
>
> As I mentioned, since the file could be big, my aim is to reduce memory
> usage when reading data from capnp database because it could be very big.
> When I read small portions of that database, I want my program not to
> consume so much memory. In the documentation, you refer to mmap usage to
> achieve this. Do you think that my approach is wrong for that purpose like
> I implemented in my code?
>
> Thanks
>
> On Wednesday, June 21, 2023 at 10:45:21 PM UTC+2 ken...@cloudflare.com
> wrote:
>
>> Hi Adrian,
>>
>> How are you measuring memory usage, exactly?
>>
>> When using mmap, measuring memory usage gets a bit complicated. The
>> kernel will load pages of the file into memory when you read then, and then
>> it is free to discard those pages at any time -- because it can always load
>> them again later if needed. But the kernel will only actually discard pages
>> if it needs the memory for something else. So if you read the entire file
>> by mmap-ing it and reading every page, and nothing else needs memory, then
>> all those pages will stay resident in memory. But this isn't really the
>> same as your program allocating memory, because, again, all those pages can
>> be freed up instantly whenever memory is needed.
>>
>> In order to fully understand what is going on you may have to dig into
>> more detailed memory stats. If your OS is just giving you a single number
>> for memory usage, it isn't telling the full story. Usually you can find a
>> bunch of different statistics if you dig in a little more.
>>
>> -Kenton
>>
>> On Wed, Jun 21, 2023 at 9:47 AM Adrian  wrote:
>>
>>> Hello
>>>
>>> I have been working on Cap'n Proto for some time to make some tests. My
>>> aim is to read the small chunks in a big serialized data to reduce the
>>> total memory consumption. For that purpose, I used memory-mapped reading
>>> and wrote a simple example to make some memory usage tests.
>>>
>>> In the tests, I realized that even if I only read the small data chunk
>>> (address) only include "address" string in itself, the total memory usage
>>> of the below test program is 512 MB in my machine (the capnp database is
>>> 2.1GB). I am wondering where I am doing something wrong. Note: I run the
>>> program only "read" mode. I called the "write" once to create capnp
>>> database.
>>>
>>> If you have any opinion, I would be very happy if you share it with me.
>>>
>>> *Proto file*
>>>
>>> --
>>> @0xa5af5d9c9e54c04a;
>>>
>>> struct Person {
>>>   name @0 :Text;
>>>   id @1 :UInt32;
>>>   email @2 :Text;
>>>   address @3 :Text;
>>> }
>>>
>>> struct AddressBook {
>>>   people @0 :List(Person);
>>> }
>>>
>>> --
>>>
>>> Source code of example
>>>
>>> --
>>>
>> #include "test.capnp.h"
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>>
>>> void writeAddressBook(int fd)
>>> {
>>> constexpr const size_t NodeNumber = 1024 * 8;
>>>
>>> ::capnp::MallocMessageBuilder message;
>>>
>>> AddressBook::Builder addressBook = message.initRoot();
>>> ::capnp::List::Builder people = addressBook.initPeople(
>>> NodeNumber);
>>>
>>> // Each string will be 128KB.
>>> constexpr const size_t size = 1024 * 128;
>>>
>>> for (int i = 0; i < NodeNumber; i++)
>>> {
>>> Person::Builder person = people[i];
>>> person.setId(i);
>>> person.setName(std::string(size, 'A').c_str());
>>> person.setEmail(std::string(size, 'A').c_str());
>>> person.setAddress("Address");
>>> }
>>>
>>> kj::VectorOutputStream output;
>>> writeMessage(output, message);
>>>
>>> auto serializedData = output.getArray();

Re: [capnproto] Memory Mapped Reading

['Kenton Varda' via Cap'n Proto]

Hi Adrian,

How are you measuring memory usage, exactly?

When using mmap, measuring memory usage gets a bit complicated. The kernel
will load pages of the file into memory when you read then, and then it is
free to discard those pages at any time -- because it can always load them
again later if needed. But the kernel will only actually discard pages if
it needs the memory for something else. So if you read the entire file by
mmap-ing it and reading every page, and nothing else needs memory, then all
those pages will stay resident in memory. But this isn't really the same as
your program allocating memory, because, again, all those pages can be
freed up instantly whenever memory is needed.

In order to fully understand what is going on you may have to dig into more
detailed memory stats. If your OS is just giving you a single number for
memory usage, it isn't telling the full story. Usually you can find a bunch
of different statistics if you dig in a little more.

-Kenton

On Wed, Jun 21, 2023 at 9:47 AM Adrian  wrote:

> Hello
>
> I have been working on Cap'n Proto for some time to make some tests. My
> aim is to read the small chunks in a big serialized data to reduce the
> total memory consumption. For that purpose, I used memory-mapped reading
> and wrote a simple example to make some memory usage tests.
>
> In the tests, I realized that even if I only read the small data chunk
> (address) only include "address" string in itself, the total memory usage
> of the below test program is 512 MB in my machine (the capnp database is
> 2.1GB). I am wondering where I am doing something wrong. Note: I run the
> program only "read" mode. I called the "write" once to create capnp
> database.
>
> If you have any opinion, I would be very happy if you share it with me.
>
> *Proto file*
>
> --
> @0xa5af5d9c9e54c04a;
>
> struct Person {
>   name @0 :Text;
>   id @1 :UInt32;
>   email @2 :Text;
>   address @3 :Text;
> }
>
> struct AddressBook {
>   people @0 :List(Person);
> }
>
> --
>
> Source code of example
>
> --
> #include "test.capnp.h"
> #include 
> #include 
> #include 
> #include 
> #include 
> #include 
> #include 
> #include 
> #include 
>
> void writeAddressBook(int fd)
> {
> constexpr const size_t NodeNumber = 1024 * 8;
>
> ::capnp::MallocMessageBuilder message;
>
> AddressBook::Builder addressBook = message.initRoot();
> ::capnp::List::Builder people = addressBook.initPeople(NodeNumber
> );
>
> // Each string will be 128KB.
> constexpr const size_t size = 1024 * 128;
>
> for (int i = 0; i < NodeNumber; i++)
> {
> Person::Builder person = people[i];
> person.setId(i);
> person.setName(std::string(size, 'A').c_str());
> person.setEmail(std::string(size, 'A').c_str());
> person.setAddress("Address");
> }
>
> kj::VectorOutputStream output;
> writeMessage(output, message);
>
> auto serializedData = output.getArray();
>
> void *dataPtr = const_cast(static_cast(
> serializedData.begin()));
> size_t dataSize = serializedData.size();
>
> size_t totalBytesWritten = 0;
> while (totalBytesWritten < dataSize)
> {
> auto numberOfBytesWritten = write(fd, static_cast(dataPtr) +
> totalBytesWritten, dataSize - totalBytesWritten);
> if (numberOfBytesWritten == -1)
> {
> throw std::runtime_error{"Error during creating capnp database"};
> }
> totalBytesWritten += numberOfBytesWritten;
> }
> }
>
> void readAddressBook(int fd)
> {
> struct stat st;
> fstat(fd, );
> size_t fileSize = st.st_size;
>
> char *mappedData = static_cast(mmap(nullptr, fileSize, PROT_READ,
> MAP_PRIVATE, fd, 0));
>
> capnp::FlatArrayMessageReader reader(kj::ArrayPtr(
> reinterpret_cast(mappedData), fileSize / sizeof(capnp
> ::word)));
>
> AddressBook::Reader addressBook = reader.getRoot();
>
> for (Person::Reader person : addressBook.getPeople())
> {
> person.getId();
> }
>
> munmap(mappedData, fileSize);
> close(fd);
> }
>
> int main(int argc, char **argv)
> {
> int fd = open("./data.bin", O_RDWR);
>
> if (!std::strcmp(argv[1], "--write"))
> {
> writeAddressBook(fd);
> }
>
> if (!std::strcmp(argv[1], "--read"))
> {
> readAddressBook(fd);
> }
>
> return 0;
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/a3192b90-a8bf-4151-84e8-0b8516d8f71bn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To 

Re: [capnproto] [Feature Request] Putting List() object at last and allow reuse of the MallocMessageBuilder

['Kenton Varda' via Cap'n Proto]

Cross-segment references are in fact expressed as a pair of a segment
number and an offset today. So with sufficiently complex allocation logic,
you could indeed allocate the side objects in a separate segment from the
list itself, and thus allow the list to be resized continuously.

But the C++ implementation, at least, was never intended to be used this
way. The main purpose of segments in C++ is to allow more space to be
allocated without relocating the existing data in memory. Each time an
allocation is unable to be satisfied by the existing segments, a new
segment is allocated, usually twice the size of the previous one.

Perhaps it would not take too much effort to expose some APIs such that you
can designate a particular list must live in its own segment, so that it
can be grown, and the segment itself can even be re-allocated in a
different location to accommodate when the list outgrows the previous
allocation. This of course means that all Reader and Builder objects
pointing into the list would be invalidated any time it grows. But I
suppose a carefully-written application could handle all that.

There is another problem with this, though, which is that "far pointers"
(that point across segment boundaries) have more overhead than in-segment
pointers. An in-segment pointer is always 8 bytes. Far pointers typically
require 8 bytes in the source segment *and* 8 bytes in the destination
segment, for a total of 16 bytes. (And when the destination segment doesn't
have space to allocate this, 16 bytes will be needed in some other segment,
for a total of 24 bytes). In a large list of small items this overhead
could be substantial.

-Kenton

On Mon, Jun 5, 2023 at 11:26 AM Jonathan Shapiro 
wrote:

> Chiming in here in relative ignorance…
>
> Offhand, it seems to me that the advantage to placing strings last is that
> you can buffer them in another segment making a single pass over the struct
> array and then appending them without a second touch. It plays fairly
> nicely with the cache, and writev() becomes your friend when the time comes
> to do your I/O.
>
> Offhand, it seems to me that you could equally well buffer the structs,
> but it depends on the on-wire representation for references.
>
> Logical truncate and physical truncate are two different things. I’d have
> to look at the wire representation to be sure, but I suspect it’s okay to
> truncate an already buffered array in the sense that the recipient will not
> be told about the (now garbage) extra bytes. Not efficient from a
> transmission perspective, and I can’t think of a good use case.
>
> Kevin: I have two ignorant design questions here:
>
> 1. Given that messages are already divided into segments, did you consider
> an on-wire representation for references taking the form segment::offset
> coupled with an in-memory segment table somewhere? I expect you did, and
> I’m wondering what the issues are/were? Wide adoption of writev() is
> relatively recent, which seems relevant here, but using a byte offset on
> the wire precludes both out-of-order segment serialization and out-of order
> segment writes.
>
> 2. If there were a reason to, does the current API expose the on-wire
> reference representation? Is the current representation effectively part of
> the API?
>
>
> Jonathan
>
>
>
> On Mon, Jun 5, 2023 at 7:32 AM 'Kenton Varda' via Cap'n Proto <
> capnproto@googlegroups.com> wrote:
>
>> Hi Gokul,
>>
>> There's currently no way to "inline" one struct into another.
>>
>> The fundamental problem you have here is this: Say you have a list
>> containing pointer values, and the list is actually at the end of the
>> message. So, you can resize it without fragmentation. You add one new
>> element. Then you populate the element, by allocating the pointer values
>> that go into it. These new allocations are added to the end of the message,
>> therefore the list is no longer at the end, and so can no longer be resized
>> without fragmentation. Yes, in theory, if you could remove all of the
>> objects allocated past the end of the list, then you could perhaps resize
>> it again. But, then you'd have to bring those object back again, added to
>> the end of the list. Each time you added a new element, the number of
>> objects that you are rolling back and forward every time increases, so the
>> overall time complexity ends up being O(n^2).
>>
>> You might want to consider an alternative design: Instead of a list,
>> maybe you want a stream. In a streaming approach, you would append each
>> item to your byte stream as a new encoded message. The down side of this
>> approach is that you cannot easily jump to a random location in the list;
>> you must parse the items in order. Bu

Re: [capnproto] [Feature Request] Putting List() object at last and allow reuse of the MallocMessageBuilder

['Kenton Varda' via Cap'n Proto]

Hi Gokul,

There's currently no way to "inline" one struct into another.

The fundamental problem you have here is this: Say you have a list
containing pointer values, and the list is actually at the end of the
message. So, you can resize it without fragmentation. You add one new
element. Then you populate the element, by allocating the pointer values
that go into it. These new allocations are added to the end of the message,
therefore the list is no longer at the end, and so can no longer be resized
without fragmentation. Yes, in theory, if you could remove all of the
objects allocated past the end of the list, then you could perhaps resize
it again. But, then you'd have to bring those object back again, added to
the end of the list. Each time you added a new element, the number of
objects that you are rolling back and forward every time increases, so the
overall time complexity ends up being O(n^2).

You might want to consider an alternative design: Instead of a list, maybe
you want a stream. In a streaming approach, you would append each item to
your byte stream as a new encoded message. The down side of this approach
is that you cannot easily jump to a random location in the list; you must
parse the items in order. But maybe that's OK for your use case.

A third approach could combine the two: You could create a stream of
messages, and separately maintain an index into that stream. The index
would store the starting byte offset of each message in the stream, so you
can seek to it. The elements of the index are fixed-size (just an integer),
so you could maintain the index as a Cap'n Proto List(UInt64) or the like.

-Kenton

On Wed, May 31, 2023 at 10:14 AM Gokul Rajiv 
wrote:

> Hi Kenton,
>
> As a follow up to the case of structs containing pointer/non-primitive
> types, might there currently exist workarounds to being able to resize
> lists of these structs (like the one below) at the back of a message:
> perhaps removing the allocated non-primitive data or embedding them inline
> in the struct?
>
> ```
> struct FourPoints {
> pt1 @0 :import "vector2.capnp".Vector2f; # bottom-left
> pt2 @1 :import "vector2.capnp".Vector2f; # bottom-right
> pt3 @2 :import "vector2.capnp".Vector2f; # top-right
> pt4 @3 :import "vector2.capnp".Vector2f; # top-left
> }
>
> # the size of TagDetection is known
> struct TagDetection {
> id @0 :UInt32;
> hammingDistance @1 :UInt8;
> tagSize @2 :Float32;
> areaPixel @3 :Float32; # number of pixels the tag occupies in the image
>
> tagCenter @4 :import "vector2.capnp".Vector2f;
>
> # corners
> pointsPolygon @5 :FourPoints;
> pointsUndistortedPolygon @6 :FourPoints;
>
> poseInCameraFrame @7 :import "se3.capnp".Se3; # in RDF camera frame
> }
> ```
>
> Of course, I could manually list the primitive fields of the constituent
> non-primitive types of the struct, but I'm trying to avoid having to do
> this.
>
> - Gokul
> On Monday, December 12, 2022 at 9:22:22 PM UTC+8 Hui min wrote:
>
>> Hi Kenton,
>>
>> Got it, it makes sense. So if I remove `labelString` data from the
>> definition, I could use truncate function to operate my detections @4
>> :List(Detection2d); object. Thanks!
>>
>> On Tuesday, 6 December 2022 at 10:43:17 pm UTC+8 ken...@cloudflare.com
>> wrote:
>>
>>> Hi Hui,
>>>
>>> Again sorry for the slow reply.
>>>
>>> In fact, the functionality you suggest does exist today, in the form of
>>> `capnp::Orphan::truncate()`, which can be used to resize a list
>>> (truncate *or* extend), doing so in-place if possible. If the list is at
>>> the end of the message, it's possible it can be extended in-place. To use
>>> it, you would do:
>>>
>>> ```
>>> auto orphan = reader.disownDetections();
>>> orphan.truncate(newSize);
>>> reader.adoptDetections(kj::mv(orphan));
>>> ```
>>>
>>> HOWEVER, there is a problem that might apply to your sample use case:
>>> Your `Detection2d` struct contains `labelString @1 :Text`, which is a
>>> pointer type. When you set this field to a string value, the string is
>>> allocated at the end of the message. This means your list is *not* at the
>>> end of the message anymore, so you are no longer able to resize the list to
>>> make it longer. To be able to extend your list, you will need the list to
>>> contain only primitive types, so that it stays at the end of the message.
>>>
>>> -Kenton
>>>
>>> On Sat, Nov 5, 2022 at 9:16 PM Hui min  wrote:
>>>
 Hi Cap'n Proto Team,

 Thanks for the amazing tool you have created so far.

 Understand that with the current design of arena style memory
 allocation, we cannot simply reuse the message, and re-init List() object
 to send it again. This will cause the message to grow every time we send
 (memory leak).

 However, sending variable length data is still pretty common practice.
 If we have to reallocate a brand new heap for new message, it is quite
 wasteful. In most cases however, the variable length list is 

Re: [capnproto] Rules for evolving union members

['Kenton Varda' via Cap'n Proto]

Hi Andrew,

Yes, you got it right. Annoyingly you will have to keep that unused Void in
there, but at least it doesn't take any space on the wire.

-Kenton

On Mon, May 1, 2023 at 4:35 PM 'Andrew Sun' via Cap'n Proto <
capnproto@googlegroups.com> wrote:

> Suppose I have a schema that looks like this:
>
> struct Foo {
> union {
> bar @0 :Void;
> baz @1 :UInt64;
> # ...
> }
>
> # ...
> }
>
> Now suppose I want to add information to union member bar, or I want to
> add more fields to baz. What options do I have to do this in a forwards-
> and backwards-compatible manner?
>
> Would this be the "correct" way to evolve the schema?
>
> struct Foo {
> union {
> bar :group {
> unused @0 :Void;
> foo @2 :UInt32;
> bar @3 :OtherStruct;
> # ...
> }
> baz :group {
> a @1 :UInt64;
> b @4 :Text;
> # ...
> }
> # ...
> }
>
> # ...
> }
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/b4a83e9d-9145-4127-b9a6-aefc9500c1b2%40app.fastmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnRTitSbn7ObuXjLqLOaBeHo9LaMtOqQ_w1y3X7W2NHhg%40mail.gmail.com.

Re: [capnproto] Re: Correctly revoking capabilities holding onto resources

['Kenton Varda' via Cap'n Proto]

It seems like what's needed here is a way to register a callback on
`kj::Canceler` which is invoked when cancel() is called. Then the
MembraneHook can register interest in being notified of cancellation,
wherein it can permanently disable itself.

Actually, there is a tricky way this can be done now:

canceler.wrap(kj::Promise(kj::NEVER_DONE).attach(kj::defer([]() {
  // This code runs when canceler.cancel() is called.
}));

This is kind of hacky, but it's enough to build what you want without
having to modify kj::Canceler. That said, modifying kj::Canceler directly
is also an option here.

Meanwhile, perhaps `MembraneHook::getCanceler()` should itself throw an
exception if cancellation has already occurred, so that we don't need a
separate `getRevocationReason()`?

Alternatively, maybe `kj::Canceler` itself should have a
`cancelPermanently()` method which not only cancels current promises but
also all promises added in the future? Plus some way to query if it has
been permanently canceled...

-Kenton

On Thu, Apr 13, 2023 at 3:39 PM Rowan Reeve  wrote:

> Hi Kenton,
>
> I've been digging around the last couple days and the approach of adding 
> *MembranePolicy::getCanceler()
> -> kj::Maybe* seems to be OK when it comes to getting rid
> of outstanding requests from within the *MembraneRequestHook *and
> *MembraneHook*, but I've run into some snags emulating the behaviour of
> *MembranePolicy::onRevoked()* but synchronously:
>
> Including just the above there is no way to determine whether a membrane
> has been revoked nor the exception to raise when attempting new requests. I
> have added two new functions to address this (which so happen to allow for
> backward compatibility):
>
>- *MembranePolicy::isRevoked() -> bool,* which not strictly necessary,
>though aids comprehension, as next function could be made to return a
>*kj::Maybe*.
>- *MembranePolicy::getRevocationReason() -> kj::Exception*, which is
>expected to throw if called when *isRevoked()* would return *false*.
>
> This allows us to reject not just outstanding requests but *future* requests
> (without relying on *onRevoked()*), where the new request promises are
> substituted with promises rejected given the exception provided by
> *MembranePolicy::getRevocationReason()*. This comes with the shortcoming
> that because we are no longer *notified* about revocation inside the
> hooks, we cannot permanently revoke existing capabilities (see *MembraneHook::
> MembraneHook* where we replace the wrapped capability with
> *newBrokenCap()*). This causes a knock-on effect where if the policy is
> unrevoked, then all the membraned capabilities which were previously
> rejecting new requests will suddenly become active again, which I don't
> think is desirable behaviour.
>
> The only thing I can think of to make this work like
> *MembranePolicy::onRevoked()* would be to register a callback, but then
> we'd need to return some sort of RAII subscription so that our hooks don't
> get called after being dropped by a client, for which I don't think KJ has
> any infrastructure. So *MembranePolicy* ends up with something like the
> following new virtual functions:
>
> *MembranePolicy::getCanceler() -> kj::Maybe* // Wraps
> requests made in hooks
> *MembranePolicy::onRevoked(kj::Function&& callback)
> -> kj::Subscription*
>
> The following diff shows what I've got so far (as per first suggested
> implementation):
> https://github.com/capnproto/capnproto/compare/master...Zentren:capnproto:sync-membrane-policy
>
> Any thoughts?
>
>
> Thanks,
>
> Rowan Reeve
>
> On Wednesday, April 5, 2023 at 8:28:18 PM UTC+2 ken...@cloudflare.com
> wrote:
>
> Hi Rowan,
>
> Right, MembranePolicy today uses exclusiveJoin() to effect an asynchronous
> revocation. The problem with such asynchronous revocation is that it
> provides no explicit guarantee of when it's safe to assume revocation has
> occurred. To use this safely, you probably need to detect when your
> objects' destructors are eventually called, when their refcount reaches
> zero, before you can assume they are no longer used.
>
> What I'm saying is we should probably add a new feature to MembranePolicy
> that uses kj::Canceler instead of exclusiveJoin(). kj::Canceler allows
> synchronous revocation. I think this should probably be a feature of
> MembranePolicy; I agree it's annoying to force applications to do it
> manually. Perhaps we could deprecate the old exclusiveJoin() approach, too,
> as the synchronous approach seems strictly better.
>
> Note that doing something like `onRevoked.then([]() { canceler.cancel();
> })` doesn't really solve anything, since the `.then()` runs asynchronously,
> so once again you have no guarantee when it will take effect.
>
> -Kenton
>
> On Thu, Mar 30, 2023 at 4:25 PM Rowan Reeve  wrote:
>
> Hi Kenton,
>
> No stress, your time is given freely and I appreciate it.
>
> Your suggestion makes sense to allow an immediate method of cancelling
> outstanding requests wrapped 

Re: [capnproto] PackedMessageReader segfault for larger buffer sizes

['Kenton Varda' via Cap'n Proto]

Hi Jonathan,

I think the problem is with this function:

std::shared_ptr uda_capnp_deserialise(const char*
bytes, size_t size)
{
// ArrayPtr requires non-const ptr, but we are only using this to read
from the bytes array
kj::ArrayPtr
buffer(reinterpret_cast(const_cast(bytes)), size);
kj::ArrayInputStream in(buffer);

return std::make_shared(in);
}

Here, you are returning a PackedMessageReader that is wrapping an
ArrayInputStream, but the ArrayInputStream is allocated on the stack, so is
no longer valid once the function returns. The stream needs to remain valid
until the PackedMessageReader is destroyed. Cap'n Proto can actually lazily
read segments of the message from the stream; it doesn't read the entire
thing in the constructor.

-Kenton

On Thu, Apr 6, 2023 at 3:26 PM Jonathan Hollocombe 
wrote:

> Hi,
>
> I am using Cap'n Proto to serialise a data tree which is then sent as a
> blob to a client to be unpacked. I was using something similar to:
>
> kj::VectorOutputStream out;
> capnp::writePackedMessage(out, builder);
> auto arr = out.getArray();
> char* buffer = (char*)malloc(arr.size());
> memcpy(buffer, arr.begin(), arr.size());
>
> and
>
> kj::ArrayPtr 
> buffer(reinterpret_cast(const_cast(bytes)),
> size);
> kj::ArrayInputStream in(buffer);
> capnp::PackedMessageReader reader{in};
>
> This is working for small tree sizes but segfaults when I get to larger
> number of elements.
>
> I've found a solution that works by switching to messageToFlatArray and
> FlatArrayMessageReader but this dramatically increases the amount of data
> that needs to be sent, especially for small trees.
>
> I've also found that writing the buffer to a file and then using
> PackedFdMessageReader also works but i'd rather avoid creating files
> unnessarily.
>
> Is there a way to use the PackedMessageReader with a byte buffer?
>
> Cheers,
> Jonathan.
>
> p.s. my example code can be found at:
> https://github.com/jholloc/capnp_test
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/ae6cafe1-f6f8-4eca-bab2-9ea3e73c3c8fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmg97g9Q4GWJ_-Svtpd6G%3DvpUZ39r6TrZcZ65jNLn8ODQ%40mail.gmail.com.

Re: [capnproto] Re: Correctly revoking capabilities holding onto resources

['Kenton Varda' via Cap'n Proto]

Hi Rowan,

Right, MembranePolicy today uses exclusiveJoin() to effect an asynchronous
revocation. The problem with such asynchronous revocation is that it
provides no explicit guarantee of when it's safe to assume revocation has
occurred. To use this safely, you probably need to detect when your
objects' destructors are eventually called, when their refcount reaches
zero, before you can assume they are no longer used.

What I'm saying is we should probably add a new feature to MembranePolicy
that uses kj::Canceler instead of exclusiveJoin(). kj::Canceler allows
synchronous revocation. I think this should probably be a feature of
MembranePolicy; I agree it's annoying to force applications to do it
manually. Perhaps we could deprecate the old exclusiveJoin() approach, too,
as the synchronous approach seems strictly better.

Note that doing something like `onRevoked.then([]() { canceler.cancel();
})` doesn't really solve anything, since the `.then()` runs asynchronously,
so once again you have no guarantee when it will take effect.

-Kenton

On Thu, Mar 30, 2023 at 4:25 PM Rowan Reeve  wrote:

> Hi Kenton,
>
> No stress, your time is given freely and I appreciate it.
>
> Your suggestion makes sense to allow an immediate method of cancelling
> outstanding requests wrapped inside a membrane. After a look over
> *membrane.c++*, I do not see a *kj::Canceller* in use, so I presume this
> is done using *kj::Promise::exclusiveJoin.* I think I see three scenarios
> being dealt with when *kj::MembranePolicy::onRevoked* resolves:
>
>1. Existing requests are eventually rejected, but the underlying call
>path might still run depending on exclusive join resolution order (i.e. it
>will run if made before *onRevoked *was resolved). [1]
>
> 
>[2]
>
> 
>2. New requests against a capability obtained before revocation are
>rejected. [1]
>
> 
>3. New requests against a capability obtained after revocation
>(replaced with a dummy) are rejected.[1]
>
> 
>
> I think requests from (1) can be immediately cancelled given access to a
> *kj::Canceller* wrapping all membraned requests. I think given the dummy
> capability injected in (3), those requests are safely rejected as-is. I
> however have a concern with (2); is it guaranteed that these new requests
> will be resolved *after* *onRevoked* is processed? I'd presume requests
> would land up on the event queue in-order, but I just wonder if there could
> be any race conditions involved. *If* it is all processed in-order, is it
> then also safe to assume that *kj* will eagerly evaluate a 
> onRevoked*.then([this]()
> { canceller.cancel(); }* relying on the result of *onRevoked, *i.e. that *this
> *is still safe to use in the *MembraneHook* and/or *MembraneRequestHook*?
>
> It's a pity that the user would need to be responsible for both manually
> cancelling outstanding requests in addition to rejecting the promises
> exposed by *kj::MembranePolicy::**onRevoked* (unless I'm missing
> something). I wonder, it seems like *kj::MembranePolicy::onRevoked* seems
> to be intended to produce promises from a *kj::ForkedPromise* under the
> hood, which itself seems to have been done as a convenience as this
> provides a single-producer/multi-consumer interface to this revocation
> "event", and *kj::Promise::exclusiveJoin* already existed to reject
> calls. Could another single-producer/multi-consumer *protected* interface
> be exposed by *kj::MembranePolicy* which handles all this inline, i.e.
> without going to the event loop but leaving the public interface unchanged?
>
> Given your current and future feedback, could I raise an issue and look
> into creating a draft PR on GitHub to start exploring the change that
> you've suggested? I will probably only get to writing any code from the
> 10th of April, so further discussion can occur here and/or on the issue in
> the meantime (whichever is preferred).
>
> Look forward to hearing from you,
>
> Rowan Reeve
>
> On Monday, March 27, 2023 at 4:43:51 PM UTC+2 ken...@cloudflare.com wrote:
>
>> Hi Rowan,
>>
>> Sorry for the slow reply, my inbox is overloaded as always.
>>
>> Indeed, since the `onRevoked` mechanism is triggered by a promise, the
>> actual revocation and cancellation occurs asynchronously. It's possible
>> that some other promise will be queued in between the point where you
>> resolve the revoker promise and when the revocation actually takes effect.
>>
>> kj::Canceler has better behavior, in that all cancellation happens
>> synchronously. But, capnp::Membrane does not currently use that. I have
>> myself hit this a couple 

Re: [capnproto] Re: CapnProto vs gRPC/Protobuf differences - semantics and use case

['Kenton Varda' via Cap'n Proto]

Hi Jonathan,

On Fri, Mar 17, 2023 at 2:10 PM Jonathan Shapiro 
wrote:

> I missed one under semantics:
>
>- capn-proto structs are defined as reference (pointer) types, while
>protobuf message types appear to be value types.
>
> Does capn-proto support the case where a single struct is referenced from
> multiple places? That is: does it support graphs as messages?
>

I think you might be confusing semantics vs. encoding details here. Structs
are *encoded* using a pointer that points to the content located elsewhere
in the message buffer. However, they nevertheless behave like value types.
The semantics are just about exactly the same as in Protobuf.

Cap'n Proto does not support graphs. Given the use of pointers, it may be
obvious how graphs would be encoded, if we supported them. The problem with
graphs is that they make so much else in the implementation vastly more
complicated. For example, say I do `message1.setFoo(message2.getFoo())`,
where `foo` has a struct type. We have to copy `foo` from one message
buffer into another. With trees, this is a trivial recursive operation. But
if graphs are allowed, now we must maintain a lookup table to remember
which pointers we've already followed. Moreover, if on the next line I do
`message1.setBar(message2.getBar())`, and it turns out `foo` and `bar` both
pointed to a common third object, how do we make sure we don't make a
redundant copy of that? It seems we now have to maintain a mapping table
long-term for any pair of messages for which copies have occurred.

On the use case front, it seems to me that the two are optimized for
>> different situations:
>>
>>- The gRPC+protobuf encoding scheme is optimized for use over lower
>>bandwidth links, but embeds the assumption that decoding upon receipt will
>>proceed linearly and to completion (because random access isn't
>>straightforward).
>>- The capn-proto encoding scheme is optimized for local area RPC
>>and/or out-of-process plugins, where communication bandwidth isn't much of
>>a limiting factor but efficient transmission (perhaps even by mmap) 
>> matters.
>>
>> Note that Cap'n Proto's serialization is not primarily designed for RPC
at all, and indeed most users use the serialization but not the RPC. The
serialization's biggest wins come when used as a format for large files
that are read using mmap().

When it comes to RPC, the serialization might lend itself nicely to
communications via shared memory, but I'm not sure if anyone has actually
tried that (yet).

I would not necessarily say that protoobuf is "optimized" for low
bandwidth. When using a low-bandwidth link, I would highly recommend
applying compression to either format, which will have greater impact than
Protobuf's encoding techniques (and will narrow the gap created by those
techniques).

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkWkAEZLVH2zOvywB0pf1-56WMfviQf47E1xBa4Thi%3DmA%40mail.gmail.com.

Re: [capnproto] Re: Correctly revoking capabilities holding onto resources

['Kenton Varda' via Cap'n Proto]

Hi Rowan,

Sorry for the slow reply, my inbox is overloaded as always.

Indeed, since the `onRevoked` mechanism is triggered by a promise, the
actual revocation and cancellation occurs asynchronously. It's possible
that some other promise will be queued in between the point where you
resolve the revoker promise and when the revocation actually takes effect.

kj::Canceler has better behavior, in that all cancellation happens
synchronously. But, capnp::Membrane does not currently use that. I have
myself hit this a couple times and ended up using hacks like you suggest.

Perhaps we should extend MembranePolicy with `getCanceler()` that returns
`kj::Maybe`. If non-null, the canceler wraps all promises
and capabilities passed through the membrane.

-Kenton

On Mon, Mar 27, 2023 at 7:35 AM Rowan Reeve  wrote:

> I've added an ugly unit test to a branch on my GitHub to illustrate:
>
>
> https://github.com/capnproto/capnproto/compare/master...Zentren:capnproto:membrane_issue?expand=1#diff-49ad79a4fffcbe88fcd8681ec67d49f5f6e5fc9010961c1b10ef1b462f0e957eR477
>
> Note line 477 in *c++/src/capnp/membrane-test.c++* where I'd expect the
> request to have been cancelled as per membrane policy *onRevoked()* docs
> ("all outstanding calls cancelled"). Looking at the behavior, it seems like
> chained promises in the request are not cancelled as part of this (only the
> initial *call(CallContext)* IF we have not yet entered its body).
>
>
> Thanks,
>
> Rowan Reeve
> On Wednesday, March 15, 2023 at 3:42:39 PM UTC+2 Rowan Reeve wrote:
>
>> Hi Kenton,
>>
>> I am encountering a problem where capabilities acting as views over some
>> resources are intermittently causing segfaults. The capability is wrapped
>> using *capnp::membrane* given a membrane policy where the promise
>> returned by *onRevoked* can be rejected on-demand via a synchronous
>> reject function (a kj::PromiseFulfillerPair is used to do this).
>>
>> The resources may be destroyed together at any time, whereby the membrane
>> managing the capabilities accessing the resource states is revoked.
>> However, this does not seem to be an instantaneous operation (presumably
>> due to revocation being managed by a promise), and I have encountered the
>> following issue as a result:
>>
>> Unresolved requests made before the membrane policy has been revoked and
>> where the resource has since been destroyed are not cancelled but will
>> rather resolve, accessing invalid memory.
>>
>> The workaround I have found to address this issue is to add a flag and a
>> *kj::Canceller* to the capability implementations whereby new requests
>> are rejected if the flag is set, and in addition when the flag is first
>> set, the canceler cancels all returned promises in cases where a chained
>> promise was returned rather than *kj::READY_NOW*. However, this is very
>> ugly and necessitates keeping around references to the capability
>> implementations before they are converted to *::Client* objects (so that
>> we can set that flag). I'm thinking that surely there has to be a better
>> way I have not considered.
>>
>> Do you have any thoughts on a better solution to this problem? If needed,
>> I can try create a minimal reproducible example to illustrate.
>>
>> In case it matters, OS is Ubuntu 20.04 and capnp version is 8.0.0, both
>> currently contained by my production environment.
>>
>> Thank you for your time,
>>
>> Rowan Reeve
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/2d126940-b82e-4ef8-9f41-304d8a23c97cn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmWFtAf3XzqQgm-rZ76H_c5J3XB_p1YNLBT1pHkkZPqiQ%40mail.gmail.com.

Re: [capnproto] Ada implementation

['Kenton Varda' via Cap'n Proto]

Hi Ralf,

Nope, I don't think I've heard of anyone working on an Ada implementation
before!

-Kenton

On Tue, Feb 21, 2023 at 6:02 AM Ralf Hubert  wrote:

> Hi,
>
> I'm thinking about a Ada implementation for Cap'n Proto.
> Do you know if there has been already some work done for Ada?
>
> BR
> Ralf
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/6e39c987-163e-4720-967b-05992a2339aen%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn%2Bm5DG_rkuQSM3s9szXu11fhr71Ue_ft6uKTqUt06QqA%40mail.gmail.com.

Re: [capnproto] Schema Mismatch Between Client and Server

['Kenton Varda' via Cap'n Proto]

Hi Matt,

Your idea would work initially, but if an @3 field were ever added, it
could end up incompatible.

Instead, assuming `Internal` is a struct type, you can instead declare
`internal` to have type `AnyPointer`:

struct Event {
  union {
foo @0 : Foo;
bar @1 : Bar;
internal @2: AnyPointer;
  }
}

Since `Internal` is a struct, a field of type `Internal` is represented as
a pointer. Since `AnyPointer` is also a pointer, it'll produce the same
layout.

At the risk of over-engineering, another option would be to use generics:

struct Event(InternalType) {
  union {
foo @0 : Foo;
bar @1 : Bar;
internal @2: InternalType;
  }
}

This way, you and your consumers can actually use exactly the same
definition of `Event`. Your consumers would use `Event<>` (which is
equivalent to `Event`). In your internal code with
knowledge of the internal type, you'd use `Event`.

-Kenton

On Tue, Feb 14, 2023 at 1:21 PM Matt Stern  wrote:

> Hi Capnp folks,
>
> I have a producer that writes events to a queue. The schema looks like:
>
> struct Event {
>   union {
> foo @0 : Foo;
> bar @1 : Bar;
> internal @2 : Internal;
>   }
> }
>
> There are downstream consumers of this queue that are meant to ignore the
> "internal" field -- in fact, we don't even provide the Internal struct
> definition to them.
>
> The downstream consumers have had a lot of log spam lately because they
> use this stripped down schema to generate code:
>
> struct Event {
>   union {
> foo @0 : Foo;
> bar @1 : Bar;
>   }
> }
>
> and then handle the union with a switch/case like so:
>
> switch (reader.which()) {
>   case FOO: return handle(reader.getFoo());
>   case BAR: return handle(reader.getBar());
>   default: return logError("A new message type exists that we don't know
> about!");
> }
>
> The default case in the switch makes sense -- if a new message type
> appears from the producer, we should surface that in some way so the
> consumers know to update their schemas and handle the new message. However,
> we don't want to trigger this case for the known Internal type, which
> consumers don't need to handle.
>
> I am thinking of giving the consumers a new schema that looks like:
>
> struct Event {
>   union {
> foo @0 : Foo;
> bar @1 : Bar;
> internal @2: Void;
>   }
> }
>
> so they can explicitly ignore the internal messages but still log errors
> for other new messages that may appear.
>
> My question boils down to: Is it safe for the producer to assign one type
> (struct Internal) to field 2 but the consumer to assign another type
> (Void)? Or would this cause issues?
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/f94fe404-b080-4654-bfc6-c578c411cf55n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DDYGk7NFjajnw4j8J%2BZM%3Dto-_ur92zL%2BqmovodO7oQug%40mail.gmail.com.

Re: [capnproto] Incomplete compile-time Zig implementation

['Kenton Varda' via Cap'n Proto]

Hmm, it's interesting that you're allowed to read non-zig source files from
disk in "comptime" but not allowed to execute external binaries or call
into non-zig libraries. If the capnp file contains an import statement, are
you allowed to open whatever file it references? If so that sounds like you
basically have arbitrary (read-only?) filesystem access?

-Kenton

On Tue, Jan 31, 2023 at 2:18 AM Yorhel  wrote:

> Hi Kenton,
>
> The project comes with a whole new parser and layouting implementation.
> I have some tests[1] for complicated struct/union layouts, but you're
> absolutely right that this needs much more rigorous testing and an
> enormous disclaimer - both of which I had planned had I continued.
>
> It's not possible to invoke external programs or call into other
> languages from Zig comptime[2], that would require a separate build
> step.  One idea I considered is to have the parser recognize the
> comments generated by `capnp -o capnp` and use them for validation, so
> that it's at least easier to get that extra assurance. Loading a file in
> schema.capnp format could also be done, of course, but then one might as
> well introduce a separate build step.
>
> 1)
> https://code.blicky.net/yorhel/capnzig/src/commit/5618819e01c8dda38962b986e4107d7d20e863f5/src/schema.zig#L812
>
> 2) Okay, Zig's comptime being turing complete one could potentially
> write a C++ interpreter and run capnp's layouting code during Zig
> compilation, but let's not go there. :)
>
> Yorhel.
>
> On 2023-01-30, 'Kenton Varda' via Cap'n Proto wrote:
> > Hi Yorhel,
> >
> > Cool stuff. It's neat that Zig can run code at compile time.
> >
> > Question, though: are you calling the C++ parser library to parse Cap'n
> > Proto schema language, or did you write a whole new parser for it? If the
> > latter, I'm worried as the algorithm to decide the layout of structs is
> > quite complex, especially when unions of groups are involved. If you are
> > re-implementing that, it would be pretty easy to get wrong leading to
> > subtle incompatibilities. Is it possible to invoke the parser from the
> C++
> > implementation as a library, to convert from capnp IDL into schema.capnp
> > format, which could then be interpreted by your implementation?
> >
> > -Kenton
> >
> > On Thu, Jan 12, 2023 at 6:06 AM 'Yorhel' via Cap'n Proto <
> > capnproto@googlegroups.com> wrote:
> >
> > > Hi list,
> > >
> > > I've been working on a Cap'n Proto implementation for the Zig language.
> > > This implementation is unusual in that it leverages the language's
> > > compile-time code execution capabilities to read a schema file -
> written
> > > in Cap'n Proto's schema language, no separate compilation step required
> > > - and then make the interfaces defined in the schema available to
> > > library users.
> > >
> > > I wasn't 100% sure if this was going to work, but I now have a parser,
> > > struct layouting algorithm and some rudimentary experiments with a
> > > type-safe struct read/write API that compiles down to code that
> *should*
> > > be just as efficient as if a code generation step was involved.
> > >
> > > Sadly, now that I've convinced myself that this might actually work
> out,
> > > I've lost the motivation to do the remaining 80% of the work and turn
> it
> > > into a usable library. So instead I'm publishing the unfinished product
> > > with the hope that it might inspire or prove useful to someone. And,
> who
> > > knows, maybe I'll continue to work on it some time in the future.
> > >
> > > The code is at https://code.blicky.net/yorhel/capnzig
> > >
> > > Documentation's kind of lacking - the code is commented here and there,
> > > but making sense of everything may take effort. I'll be around to
> answer
> > > questions if there are any.
> > >
> > > Here's hoping we'll have a mature Cap'n Proto implementation for Zig at
> > > some point, regardless of which code generation strategy is taken.
> > >
> > > Yorhel.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Cap'n Proto" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an
> > > email to capnproto+unsubscr...@googlegroups.com.
> > > To view this discussion on the web visit
> > > https://groups.google.com/d/msgid/capnproto/Y7/3v1VZfkFIFO5s%40gmai021
> .
> > >
> >
> > --
> > You received this 

Re: [capnproto] Incomplete compile-time Zig implementation

['Kenton Varda' via Cap'n Proto]

(Sorry for the very slow reply, my inbox is a mess.)

-Kenton

On Mon, Jan 30, 2023 at 6:09 PM Kenton Varda  wrote:

> Hi Yorhel,
>
> Cool stuff. It's neat that Zig can run code at compile time.
>
> Question, though: are you calling the C++ parser library to parse Cap'n
> Proto schema language, or did you write a whole new parser for it? If the
> latter, I'm worried as the algorithm to decide the layout of structs is
> quite complex, especially when unions of groups are involved. If you are
> re-implementing that, it would be pretty easy to get wrong leading to
> subtle incompatibilities. Is it possible to invoke the parser from the C++
> implementation as a library, to convert from capnp IDL into schema.capnp
> format, which could then be interpreted by your implementation?
>
> -Kenton
>
> On Thu, Jan 12, 2023 at 6:06 AM 'Yorhel' via Cap'n Proto <
> capnproto@googlegroups.com> wrote:
>
>> Hi list,
>>
>> I've been working on a Cap'n Proto implementation for the Zig language.
>> This implementation is unusual in that it leverages the language's
>> compile-time code execution capabilities to read a schema file - written
>> in Cap'n Proto's schema language, no separate compilation step required
>> - and then make the interfaces defined in the schema available to
>> library users.
>>
>> I wasn't 100% sure if this was going to work, but I now have a parser,
>> struct layouting algorithm and some rudimentary experiments with a
>> type-safe struct read/write API that compiles down to code that *should*
>> be just as efficient as if a code generation step was involved.
>>
>> Sadly, now that I've convinced myself that this might actually work out,
>> I've lost the motivation to do the remaining 80% of the work and turn it
>> into a usable library. So instead I'm publishing the unfinished product
>> with the hope that it might inspire or prove useful to someone. And, who
>> knows, maybe I'll continue to work on it some time in the future.
>>
>> The code is at https://code.blicky.net/yorhel/capnzig
>>
>> Documentation's kind of lacking - the code is commented here and there,
>> but making sense of everything may take effort. I'll be around to answer
>> questions if there are any.
>>
>> Here's hoping we'll have a mature Cap'n Proto implementation for Zig at
>> some point, regardless of which code generation strategy is taken.
>>
>> Yorhel.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Cap'n Proto" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to capnproto+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/capnproto/Y7/3v1VZfkFIFO5s%40gmai021.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnHZsYcGMMtQW_Gu74uZu%2BnALdCRkL_XPZiP7x63UD-Lg%40mail.gmail.com.

Re: [capnproto] Incomplete compile-time Zig implementation

['Kenton Varda' via Cap'n Proto]

Hi Yorhel,

Cool stuff. It's neat that Zig can run code at compile time.

Question, though: are you calling the C++ parser library to parse Cap'n
Proto schema language, or did you write a whole new parser for it? If the
latter, I'm worried as the algorithm to decide the layout of structs is
quite complex, especially when unions of groups are involved. If you are
re-implementing that, it would be pretty easy to get wrong leading to
subtle incompatibilities. Is it possible to invoke the parser from the C++
implementation as a library, to convert from capnp IDL into schema.capnp
format, which could then be interpreted by your implementation?

-Kenton

On Thu, Jan 12, 2023 at 6:06 AM 'Yorhel' via Cap'n Proto <
capnproto@googlegroups.com> wrote:

> Hi list,
>
> I've been working on a Cap'n Proto implementation for the Zig language.
> This implementation is unusual in that it leverages the language's
> compile-time code execution capabilities to read a schema file - written
> in Cap'n Proto's schema language, no separate compilation step required
> - and then make the interfaces defined in the schema available to
> library users.
>
> I wasn't 100% sure if this was going to work, but I now have a parser,
> struct layouting algorithm and some rudimentary experiments with a
> type-safe struct read/write API that compiles down to code that *should*
> be just as efficient as if a code generation step was involved.
>
> Sadly, now that I've convinced myself that this might actually work out,
> I've lost the motivation to do the remaining 80% of the work and turn it
> into a usable library. So instead I'm publishing the unfinished product
> with the hope that it might inspire or prove useful to someone. And, who
> knows, maybe I'll continue to work on it some time in the future.
>
> The code is at https://code.blicky.net/yorhel/capnzig
>
> Documentation's kind of lacking - the code is commented here and there,
> but making sense of everything may take effort. I'll be around to answer
> questions if there are any.
>
> Here's hoping we'll have a mature Cap'n Proto implementation for Zig at
> some point, regardless of which code generation strategy is taken.
>
> Yorhel.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/Y7/3v1VZfkFIFO5s%40gmai021.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DFvmUpqCvqEtGAvU4gtMo5V8drrGcPHXWRuPBYc254PA%40mail.gmail.com.

Re: [capnproto] List of List using DynamicList

['Kenton Varda' via Cap'n Proto]

On Tue, Jan 24, 2023 at 6:11 PM Carlo Contavalli 
wrote:

> Hello,
>
> On Tue, Jan 24, 2023 at 1:10 PM Kenton Varda 
> wrote:
>
>> You mean SchemaLoader::getType(), right?
>>
>
> Correct.
>
>
>> case schema::Type::LIST:
>> return ListSchema::of(getType(proto.getList().getElementType(), scope));
>>
>> So the returned type should not lose the fact that the type represents a
>> list (or list-of-lists, or list-of-list-of-lists, etc.).
>>
>> Do you have some code that doesn't work as you expect, that you could
>> share?
>>
>
> When looking at the code above, the recursion seemed to pretty much return
> a type representing List(struct) rather than
> List(List(struct)),
>

I don't think that's true. If you give it a representation of
List(List(struct)), it'll return ListSchema::of(ListSchema::of(struct
type)), which seems correct?


> I am not sure iterating over the DynamicList is doing the right thing. My
> code is crashing, so assumed I was doing something wrong.
>
> Let me work on a self contained example, don't have one handy.
>

Yes, I think that would help.


> Fundamentally... Once getType() returns, I have a capnp::Type that I can
> use to build a DynamicList, which I can use to iterate over the elements.
> Iterating over the elements is moving through the outer list, and returns
> a DynamicValue. Now I need to iterate over the elements on the inner list.
>
> To go from DynamicValue to another DynamicList... use getType() passing in
> getList().getElementType() (recursing inside once), and then
> DynamicValue.of>?
>

You would use `dynamicValue.as()`, which returns a
DynamicList::Reader for the inner list, which you can then iterate over.

-Kenton

>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D_Uf-OX9jDziJDizML%3DQz8Mh0o%3DxbSq%3DOsKW9wZnJ8Rw%40mail.gmail.com.

Re: [capnproto] List of List using DynamicList

['Kenton Varda' via Cap'n Proto]

Hi Carlo,

It seems like you've figured out what everything represents and than you
need to use SchemaLoader here. I'm not sure I understand the problem you're
getting at.

On Sat, Jan 21, 2023 at 11:30 AM Carlo Contavalli 
wrote:

> I see I can use SchemaLoader.get() to get a capnp::Type out of a
> schema::Type::Reader, and from there I can get the ListSchema, and
> DynamicList. But the .get() method recurses on the Type, so it'll return
> the type of the inner struct, rather than outer List, and I suspect the
> returned DynamicList will iterate on the inner struct, rather than the
> outer list.
>

You mean SchemaLoader::getType(), right? Sure, the code "recurses" to
convert the list's element type, but it then wraps that type in
`ListSchema::of()`:

case schema::Type::LIST:
return ListSchema::of(getType(proto.getList().getElementType(), scope));

So the returned type should not lose the fact that the type represents a
list (or list-of-lists, or list-of-list-of-lists, etc.).

Do you have some code that doesn't work as you expect, that you could share?

Sorry, I don't know of any example code off the top of my head, though
perhaps there is something in schema-test.c++, schema-loader-test.c++, or
dynamic-test.c++.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkDWd2Vci%3Dn%3DOxiLuVQAr7yzuGk7mtj0d3RZHiNewg-sw%40mail.gmail.com.

Re: [capnproto] Byte Order of multi byte values

['Kenton Varda' via Cap'n Proto]

Hi Stefan,

Cap'n Proto automatically takes care of byte ordering. You do not need to
worry about this.

Generally, all modern serialization libraries will take care of endianness
automatically. The only time you need to worry about byte-swapping is if
you are reading raw bytes from the wire and interpreting them as integers
yourself, without the help of a serialization library like Cap'n Proto.

Note that Cap'n Proto uses little-endian encoding on the wire. Almost all
processors in wide use these days are little-endian, so this issue hardly
even comes up anymore. However, Cap'n Proto is designed to detect if it is
running on a big-endian architecture and, if so, it'll swap the bytes
automatically.

-Kenton

On Thu, Jan 19, 2023 at 2:59 AM Stefan Schmelz  wrote:

>
> Hey everyone,
> Do you guys know if I have to ensure Byte Order for multi byte values and
> larger integers?
> I was working on a project that did parts of the encoding themselves, so
> Big-/Little endianness would be ensured.
>
> Now I am refactoring a fork of that library for myself and wanted to
> remove anything that is unnecessary. Encoding a 64bit integer as an array
> of 8 bytes is a little excessive, so I would like to remove it.
>
> Nonetheless I still need to ensure that Linux, windows, Mac OS work on all
> hardware platforms. I did read the documentation as if capnp would ensure
> that everything will be encoded and decoded correctly on every platform.
>
> I'm improving on prior work, but try to eliminate some naive mistakes we
> made the first time. By now I have written a schema generator and a
> primitive schema registry. You can create global schema that are stored in
> a central location and symlinked into your source. The central directory
> can be converted to a git repo, so it can be shared. You can also generate
> local schema, that are stored in your sources.
>
> The original project used python to generate wrapping C++ classes to
> automatically load from byte arrays and do some validation... At that time
> nothing like this was documented by Capnproto so we did everything
> ourselves.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/CAKJF573hOOgPLSVP2u6OoQmagUmX_0T-08hqYqP1F%3DtXdvbdLg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmfubu70vBcPMjyndqhLXKwNNX-HEg%2BHuNVygK1Bz5_yA%40mail.gmail.com.

Re: [capnproto] Protocol "disembargo" message

['Kenton Varda' via Cap'n Proto]

The other one that comes to my mind off the top of my head (and is also
quite old) is Waterken:

https://waterken.sourceforge.net/

Capability people will often distinguish between CapTP-style and
"Ken-style" (from Waterken) capabilities. I don't quite remember what the
main differences are, though, except that Cap'n Proto is much closer to
CapTP.

-Kenton

On Mon, Dec 26, 2022 at 10:34 PM Jens Alfke  wrote:

>
>
> On Dec 26, 2022, at 3:28 PM, Kenton Varda  wrote:
>
>  I'm not sure what you mean about "(negative) remote capability #". There
> are no negative numbers.
>
>
> Oops, I got that from the old E documentation (erights.org.) They used
> negative numbers to identify capability IDs that were created on the “other
> side” of the connection, i.e. promises.
>
> (Speaking of which, is there any other good reading material on
> capabilities-with-RPC, besides erights.org and your own rpc.capnp?)
>
> —Jens
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DHVp9%2B37VmwS_7W5QdG4kkqQ_SDrMwptUXGfpzvytLCw%40mail.gmail.com.

Re: [capnproto] Protocol "disembargo" message

['Kenton Varda' via Cap'n Proto]

On Mon, Dec 26, 2022 at 4:42 PM Jens Alfke  wrote:

> Thanks. I then spent some time trying to figure out why this scenario
> occurs — when the local peer received the capability Carol from the remote
> one, shouldn’t it have been marked in the protocol as being a peer exported
> by the recipient?
>
> I think this can happen as follows; is this correct?
>
> 0. I’ve already sent the capability Carol to the remote peer earlier in
> the connection, so the peer has a reference to Carol.
> 1. I send an RPC call to acquire a remote capability, and allocate a
> (negative) remote capability # to it. Call it X. (I think this is what the
> protocol calls a promise?)
> 2. Before the response arrives, I pipeline some more RPC calls addressed
> to X.
> 3. I get a response to the first RPC call, identifying X as my capability
> Carol.
>
> At this point I can remap X to point to Carol, but I’ve already got some
> messages in flight addressed to remote capability X. I assume what happens
> to these is the peer just sends them back to me, substituting Carol for X,
> and then forwards my reply back to me? Thus the problem that I might send
> local messages to Carol that would arrive before the echoed messages to X
> even though I sent them later.
>

That's correct, except one minor detail: I'm not sure what you mean about
"(negative) remote capability #". There are no negative numbers. The caller
assigns a question ID to the outgoing RPC, and can identify expected
results for promise pipelining by the question ID plus a field path.


> A different architectural solution to this problem might be for a peer to
> reject an incoming request addressed to a capability that isn’t local.
> Instead it would return an error indicating “X isn’t mine, it’s your Carol,
> so forward it there”.
>

Sure, but this would have several problems:
* If the RPC system is expected to deal with it transparently, then that
implies the RPC system has to keep a copy of every message it has sent on a
promise, until either it gets a response or the promise resolves, wasting
memory in order to handle an unusual situation.

* If the RPC system is NOT expected to deal with it transparently, then the
application has to deal with it. This would compromise the abstraction, as
application code would now have to be written with awareness of which
objects reside in which vats in order to be ready for this scenario.

* In some three-party scenarios (when that is implemented), it would result
in extra network round trips. Imagine a scenario where a client talks to a
server in a remote datacenter, and the server then redirects the client to
an adjacent server. In a naive redirect scenario, the first server must
send a message to the client redirecting it, and the client must
re-send the message to the second server, all before the second server can
receive the message at all. This is an extra round trip on the
long-distance connection between the client and the two servers. With Cap'n
Proto's approach, the first server directly forwards the request to the
second server, while in parallel letting the client know to expect the
response from that server. The second server can then send the response
directly to the client. So no extra network round trip is needed.


> But either way, isn’t there still a race condition if, in a new step 2.5,
> I send a message to local Carol? This message was sent after the message to
> X but arrives before I discover X is Carol, so it’s received out of order.
>

E-order only guarantees that calls made on the same capability will be
received in order. If you make calls on two different capabilities that,
unknown to you, end up pointing to the same destination object, no
guarantees are made regarding the order in which those calls are delivered.
This has been shown to match what developers intuitively expect in practice.

 -Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQm1prYCndGsgbuHJm15UXdwK3fBvApfVM5G1%3DKbmhD9YQ%40mail.gmail.com.

Re: [capnproto] Protocol "disembargo" message

['Kenton Varda' via Cap'n Proto]

Hi Jens,

To clarify, when I say "Carol lives in Vat A, i.e. next to Alice", I am
saying that Alice and Carol are two objects living in the same process. So
a capability pointing to Carol was passed across the network and then back
again, over the same connection. In this case, we want to shorten the path
so that calls from Alice to Carol are direct local calls, not bouncing
across the network connection. This scenario does in fact come up in Level
1, as it only involves two "vats" (a "vat" is essentially one process
running Cap'n Proto; the RPC protocol is used to communicate between vats).

Embargoes are needed here for the reasons described in the comments: When
Alice discovers her promise P resolves to Carol, she may have already sent
some messages towards the promise using pipelining. Before Alice can start
making direct local calls to Carol, those messages have to cross the
network connection and reflect back, so that they are delivered first.

-Kenton

On Sat, Dec 17, 2022 at 3:18 PM Jens Alfke  wrote:

> As I peruse the RPC spec (rpc.capnp), I’m following along pretty well
> until I hit the ‘disembargo’ message (which my spellchecker keeps changing
> into ‘disembark’, but that’s another matter.)
>
> At this point I’m only interested in Level 1 functionality, two peers,
> which AFAIK is as far as the implementation is gotten.Disembargo is a level
> 1 message, and I’m having some trouble understanding it, partly because it
> seems to only be relevant at levels 2+. The description starts on line 660:
>
>   # Message sent to indicate that an embargo on a recently-resolved
> promise may now be lifted.
>   #
>   # Embargos are used to enforce E-order in the presence of promise
> resolution.  That is, if an
>   # application makes two calls foo() and bar() on the same capability
> reference, in that order,
>   # the calls should be delivered in the order in which they were made.
> But if foo() is called
>   # on a promise, and that promise happens to resolve before bar() is
> called, then the two calls
>   # may travel different paths over the network, and thus could arrive in
> the wrong order.  In
>   # this case, the call to `bar()` must be embargoed, and a `Disembargo`
> message must be sent along
>   # the same path as `foo()` to ensure that the `Disembargo` arrives after
> `foo()`.  Once the
>   # `Disembargo` arrives, `bar()` can then be delivered.
>
>
> I was following along OK until “the two calls may travel different paths
> over the network.” We’re at Level 1, so there is only one path, the socket
> between peer A and peer B.
>
> Also, in "that promise happens to resolve before bar() is called”, should
> that be "*after* bar() is called”? Because if foo resolves before, why
> would that make the foo call arrive after the bar call?
>
> If you have time, Kenton, I’d love to see the " Carol lives in Vat A, i.e.
> next to Alice.  In this case, Vat A needs to send a `Disembargo` message
> that echos through Vat B and back…” scenario broken down step-by-step. (Or
> if this situation is discussed in the old erights.org site, let me know
> the URL; I didn’t see it when I was reading through their protocol docs.)
>
> Thanks!
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/E3F6C227-23BD-41E9-A4BA-A93EE4586C49%40mooseyard.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn8z9VdU2wNeix%3DDjeOzr%3Div9C8_ej%3DbN0PJEaiMYS27Q%40mail.gmail.com.

Re: [capnproto] RPC protocol framing details?

['Kenton Varda' via Cap'n Proto]

Yes, the messages are simply written to the wire one after another. Cap'n
Proto messages in the standard serialization are self-delimiting, so
there's no need for a framing protocol.

-Kenton

On Fri, Dec 16, 2022 at 12:07 PM Jens Alfke  wrote:

> The RPC docs say:
>
> > The Cap'n Proto RPC protocol is defined in terms of Cap'n Proto
> serialization schemas. The documentation is inline. See rpc.capnp.
>
> That interface definition is beautifully commented, it’s practically
> Literate Programming, but it’s not the whole story. How are the messages
> (serialized structs) sent over the wire? Is it as simple as just writing
> one serialized buffer after another to a byte-stream, or is there an
> underlying framing protocol?
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/D26CD0E2-861D-4EA5-868F-CBDB109C%40mooseyard.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D5PTGuRdeEVyyPG5RLJCy2UFy7%3DwsxBfsu0QPepOCAZg%40mail.gmail.com.

Re: [capnproto] Build error on QNX

['Kenton Varda' via Cap'n Proto]

Hi Ze,

I haven't heard of anyone porting Cap'n Proto to QNX, but it should be
possible.

`sigaltstack()` is only used as part of `printStackTraceOnCrash()`, which
is a totally optional helper that is very OS-specific. You could start by
adding some #if/#else such that this function is empty on QNX. There's
probably a number of other places around the library where you'll need to
do similar things, but not too many, as long as you're sticking to just
libkj and libcapnp. libkj-async will probably be a lot more complicated to
support, but it's only needed for RPC -- if you only care about
serialization then you can ignore that part.

-Kenton

On Wed, Dec 7, 2022 at 12:51 AM Ze Liu  wrote:

> Hi everyone，
>
> Recently I'm working on building capnp libraries on QNX. And I faced an
> issue that kj library source code has the same header files with QNX, like
> ‘exception.h’. And also the kj library needs some interfaces that QNX
> doesn't provide, like signaltstack. For this interface, the QNX header file
> says not supported for now.
>
> I wonder is there a special version adapted with QNX? Or has anyone tried
> to use capnp on QNX?
>
> Many thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/1ea370eb-d111-402c-9fa8-81511c2ae7b4n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkmxbJrYOpk4_6XAeuPrQsT4b%3D5X3CpGPuLyRZPfCjoGQ%40mail.gmail.com.

Re: [capnproto] Server process stops accepting connections

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Sorry for the slow reply, I've been on leave.

In general, KJ async I/O objects are tied to the thread / event loop where
they were created. This means you cannot pass an AsyncIoStream between
threads. You can, however, pass a file descriptor. If you are careful, you
could tear down the AsyncIoStream wrapping your socket, pass the socket
file descriptor to another thread, and construct a new AsyncIoStream around
it there.

-Kenton

On Wed, Nov 9, 2022 at 12:19 PM Jens Alfke  wrote:

>
> On Nov 9, 2022, at 6:47 AM, 'Kenton Varda' via Cap'n Proto <
> capnproto@googlegroups.com> wrote:
>
> Have you tested whether your server is able to accept concurrent
> connections normally? E.g. if you open a connection with telnet or
> something without sending any bytes, leave that open, and then try to use
> your server, does it work?
>
>
> *Thanks! That was indeed the problem.* The telnet test showed my server
> would accept multiple connections at once, but would only authenticate one
> of them at a time, so a client that didn’t send the right handshake would
> block others.
>
> (I’m using custom code, largely copied from rpc.c++ and tls.c++, but
> substituting the SecretHandshake protocol (from Scuttlebutt) for TLS. When
> I adapted TlsConnectionReceiver::acceptLoop, I got things in the wrong
> order, so the tail-recursion back to acceptLoop didn’t occur until the
> secure handshake finished.)
>
> I’ve fixed it now. Next question: *if I want to run client connections on
> multiple threads*, I presume that after the accept-and-handshake
> finishes, I would create a new thread with an event loop, then use an
> Executor to pass the AsyncIoStream to it, and then do the magic RPC setup
> on that thread. On disconnect I’d either exit the thread or return it to a
> pool for reuse. Most of this seems like boilerplate … are there any working
> examples I could steal from?
>
> (Why would I want multiple threads when CapnP is async? Because my actual
> RPC methods do SQLite queries, which are blocking and sometimes slow, and
> I’d rather not create async wrappers for my entire complicated database
> API.)
>
> —Jens
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DZfFUQYcFzKfUtJ57KuXH5Jpbd2HqX4Hbh8wLGdW8-Hw%40mail.gmail.com.

Re: [capnproto] capnp::messageToFlatArray to pre-allocated buffer?

['Kenton Varda' via Cap'n Proto]

Hi Hui,

Sorry for the slow reply, I was on leave.

messageToFlatArray() is a convenience function for when you don't mind
allocation and copying. If you want more control over serialization, you
can use MessageBuilder::getSegmentsForOutput() to get direct pointers to
the message's underlying buffer space, without copying anything. You can
then serialize these segments as you see fit. Note that you are responsible
for tracking the segment boundaries in this case. On the receiving end,
pass the same array-of-arrays into SegmentArrayMessageReader to parse the
message. Again, this doesn't perform any allocation or copies; that's up to
you to manage.

Another option is to implement your own kj::OutputStream, and then use
capnp::writeMessage() to write a message to the stream. This will always
write the entire message using one single call to write() on the underlying
stream, passing an array-of-arrays that point into the original message
buffer -- again, no copies. You could then copy into your scratch buffer or
do whatever else you want. An advantage of this approach (using
OutputStream instead of getSegmentsForOutput()) is that writeMessage() will
also build a segment table for you and include that in the output, so on
the receiving end you can use FlatArrayMessageReader instead of
SegmentArrayMessageReader.

-Kenton

On Sat, Nov 5, 2022 at 9:29 PM Hui min  wrote:

> Hi CapnProto Team,
>
> I understand the API to obtain a flat buffer of the built message is
> capnp::messageToFlatArray.
>
> However, the current API forces you to save the buffer on a returned
> kj::Array words object. Is there a way I could pass in a
> pre-allocated buffer, and let CapnProto to use that for serialisation
> (which essentially is creating a header + concat all the segments?)?
>
> Currently, some of the API has to be implemented in this way, when
> pre-allocated buffer is present. This introduced an extra copy and
> serialisation time.
>
> https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/include/ecal/msg/capnproto/publisher.h#L147-L150
>
> A good kind of API would be like this:
>
> https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/include/ecal/msg/protobuf/publisher.h#L162
>
> Thanks in advance!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/70bbe8b2-c06d-495f-ac6d-6eedecbdd305n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkZ3uQPfobVzqZnOWsn6m-AhW3kt-tGZxAoFTvg_nK7vw%40mail.gmail.com.

Re: [capnproto] Re: Cap'n Proto CVE-2022-46149

['Kenton Varda' via Cap'n Proto]

Thanks!

On Fri, Dec 2, 2022 at 12:12 PM tripl...@gmail.com 
wrote:

> This is only relevant if you use the pypi packages. If you compile pycapnp
> with system libcapnp you should only have to update your system packages.
>

Note that, unfortunately, you also need to recompile pycapnp against the
updated system packages, since part of the change is in inlined code.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkZNFnSWA46HuHVT7OUWQ%3DquR9ubJFfEWBiFQnOudTPxw%40mail.gmail.com.

[capnproto] Cap'n Proto CVE-2022-46149

['Kenton Varda' via Cap'n Proto]

Hi capnproto,

We have a security advisory today. Although we believe few applications are
actually affected, we recommend patching anyway. You can find the details
here:

https://github.com/capnproto/capnproto/security/advisories/GHSA-qqff-4vw4-f6hx

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQm292GubUOiSV9%2BUKuq2Dv%2BDJNEfYh%2BfXhRsAGB1QCzwg%40mail.gmail.com.

Re: [capnproto] Fibers and compiling without exceptions

['Kenton Varda' via Cap'n Proto]

Hi John,

Sorry for the very slow response, I've been on leave.

Fibers won't be used unless you explicitly ask for them.

However, regardless of fibers, you cannot use a version of Cap'n Proto
compiled with exceptions enabled inside of an application compiled with
exceptions disabled. Compiling with -fno-exceptions is essentially an
ABI-breaking switch for libcapnp and libkj; you must apply the same flag
when building the library itself and the app that uses it.

So, unfortunately, it sounds like you cannot use the system-installed
capnp, which is typically built with exceptions enabled.

-Kenton

On Mon, Oct 10, 2022 at 3:36 PM John Demme  wrote:

> Hi All-
>
> I have a code base which doesn't have exceptions enabled (it's an LLVM
> code base wherein policy prevents compiling with exceptions). I'd like to
> use the Cap'n Proto (C++) RPC server but cannot use fibers without
> exceptions. I know it's possible to compile Cap'n Proto without fibers but
> I'd rather not require a custom compile for my fellow devs -- I'd rather
> let them use their system install of capnp. Is this possible?
>
> ~John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/34f283a1-8bf7-4fd8-946f-cbabf4d82bb9n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQktUHoetsA-o5%3DC-9RtX5f4KCzpcAjQTXOdQUhWrw91YA%40mail.gmail.com.

Re: [capnproto] Server process stops accepting connections

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Have you tested whether your server is able to accept concurrent
connections normally? E.g. if you open a connection with telnet or
something without sending any bytes, leave that open, and then try to use
your server, does it work?

What does your listen loop look like?

-Kenton

On Tue, Nov 8, 2022 at 1:33 PM Jens Alfke  wrote:

> I’ve got a CapnP-based server program running on a Raspberry Pi 4. It’s
> written in C++, using CapnP 0.9.1. It’s single-threaded; the main()
> function just sets up a listener, calls kj::NEVER_DONE.wait, and stays
> there.
>
> It works fine, except that every few weeks it stops accepting connections;
> it either doesn’t accept the incoming socket at all or never reads from it,
> I can’t tell which; all I see is that the client times out and gives up
> after 15 seconds. Then I have to kill and relaunch the server, after which
> it works fine again for a while.
>
> The server process itself isn’t hung; it’s in epoll waiting for events. I
> attached gdb and got this backtrace:
>
> #0  0xb6b8563c in epoll_wait (epfd=7, events=0xbecb4210, maxevents=16,
>  timeout=-1)
> #1  0x0052edcc in kj::UnixEventPort::doEpollWait (this=0x1fed420,
> timeout=-1)
> #2  0x0052e58c in kj::UnixEventPort::wait (this=0x1fed420)
> #3  0x00487744 in kj::EventLoop::wait (this=0x1fed4e0)
> #4  0x00488460 in kj::_::waitImpl (node=..., result=..., waitScope=…)
> #5  0x00488bd8 in kj::_::NeverDone::wait (this=0x72d3c0 ,
>  waitScope=…)
> …
>
>
> The last line in the log shows that it received a connection from some
> unknown IPv4 address which geolocates to Russia (this happens a few times a
> day; the server’s on my home LAN but exposes a public port and I assume
> these are random hackers looking for vulns.) This connection never did
> anything, not surprisingly since I doubt hackers are expecting anything
> other than HTTP, but the logs show the socket never closed. And sure
> enough, if I run `lsof` I see an open TCP port from that address. (Which
> has been open for at least 24 hours, strangely; doesn’t the disconnect idle
> TCP connections after 90 minutes?)
>
> I’m not sure what to do about this. I assume CapnP is capable of handling
> multiple incoming connections, so this idle socket won’t block others from
> connecting, right? But if not, then why isn’t it accepting any connections?
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/008DEC56-3B4D-4CF7-BFC3-5BECC13BE502%40mooseyard.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D6xH_pyvRGBnRpTma6_K2mU8Ogn%2BptcQ9RytARhAUrAw%40mail.gmail.com.

Re: [capnproto] Text blob too big.

['Kenton Varda' via Cap'n Proto]

For a single Text or Data blob there is a hard limit of 512MB. You can,
however, construct a message which contains multiple blobs, e.g. use
`List(Text)`. Such a message can be up to 2^64 bytes.

If I were redesigning the encoding from scratch I'd probably allow for
bigger individual blobs but there's no way to introduce them now without
breaking compatibility, unfortunately.

Protobuf theoretically supports 2GB messages but because the messages have
to be parsed upfront in O(n) time, you won't have good results with
messages of that size. Cap'n Proto, on the other hand, quite comfortably
supports multi-GB messages since you can mmap() the message and randomly
access it in O(1) time.

-Kenton

On Mon, Oct 31, 2022 at 2:21 PM Josemi  wrote:

> Hello.
>
> I need to work with an structured data that have atributes with undefined
> lenght, some of them could have GB.
>
> I have been using Protocol Buffers for it until I see that exists a hard
> limit of 2GB per message.   Then I see this Stack Overflow solution
> .
> It says that  Cap'n Proto "can support messages up to 2^64 bytes (2^32
> segments of 4GB each)"
>
> I reprogram the code and now it raise the error  *capnp.lib.capnp.KjException:
> capnp/layout.c++:1694: failed: text blob too big  * trying to set a 0.8
> GB buffer to a Data type.
>
> On Cap'n Proto the atribute seems like that:
> *file @1 :Data; # ptr[1],*
>
> And the code, in Python,  is something like:
> file = open('data_file', 'rb').read()
>
>
> Whats wrong with that.
> Cap'n Proto won't solve my problem?.
>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/8a23280b-b43b-483a-94db-7fd94bba93een%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQk0hybS%2Bc21E7NagDBQqNcCaXkTHWoPsmAT%3DiwNdHjfwg%40mail.gmail.com.

Re: [capnproto] Wikipedia "Comparison of data-serialization formats" entry

['Kenton Varda' via Cap'n Proto]

(Oops, your emails weren't going through from this address because they
were landing in the Google Groups moderation queue. I just went and cleared
it out (resulting in these dupes, sorry) and allowlisted this address so it
won't be moderated in the future.)

-Kenton

On Mon, Oct 17, 2022 at 12:08 PM Christophe Alexandre <
christophe.a...@gmail.com> wrote:

> I took the initiative to add/restore the Cap'n Proto line.
> Please tell me if anything is wrong of if more links/references are needed.
>
> Thanks
> Christophe
>
> Le vendredi 30 septembre 2022 à 01:49:22 UTC+2, ken...@cloudflare.com a
> écrit :
>
>> It used to be there, but someone removed it, with the comment "clean list
>> of less notable formats".
>>
>>
>> https://en.wikipedia.org/w/index.php?title=Comparison_of_data-serialization_formats=revision=907829979=907825303
>>
>> At the time, there was no stand-alone Cap'n Proto article. It's possible
>> the editor decided that anything not notable enough to have its own article
>> should be removed from the table.
>>
>> Since that time, it looks like someone added an article:
>>
>> https://en.wikipedia.org/wiki/Cap%27n_Proto
>>
>> So maybe it makes sense to restore the row in the table now?
>>
>> I suspect Wikipedia rules prohibit me from doing this personally.
>>
>> -Kenton
>>
>> On Thu, Sep 29, 2022 at 1:45 PM Niels Andersen 
>> wrote:
>>
>>> It would be great if someone could add Cap'n Proto to this page
>>> https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats
>>> - I don't have enough knowledge to do it.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/0480d668-00b7-4503-b9fd-836946f8035en%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/9b949d2f-a713-4af8-bcd6-d6b351da6d3en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnEOS5PuntRVQmrAVfDhp5ZN%2BbrAEEn4HmpCg5dx0v5tg%40mail.gmail.com.

Re: [capnproto] Wikipedia "Comparison of data-serialization formats" entry

['Kenton Varda' via Cap'n Proto]

Thanks.

FWIW I'd argue that Cap'n Proto should have a "no" under "Supports
references?". Although the binary encoding uses pointers, implementations
generally forbid the message structure to be anything other than
tree-shaped. You're not supposed to ever have multiple pointers pointing at
the same object, even if it's technically possible to encode, because
(among other things) supporting duplicate references and cycles would make
the copy algorithm much more complicated.

-Kenton

On Sun, Oct 16, 2022 at 5:45 AM Christophe Alexandre <
christophe.alexan...@zellij.io> wrote:

> I took the initiative to restore/add the Cap'n Proto line.
> Please tell me if everything is correct and if there are references to add.
> Thanks
> Christophe
>
>
> Le vendredi 30 septembre 2022 à 01:49:22 UTC+2, ken...@cloudflare.com a
> écrit :
>
>> It used to be there, but someone removed it, with the comment "clean list
>> of less notable formats".
>>
>>
>> https://en.wikipedia.org/w/index.php?title=Comparison_of_data-serialization_formats=revision=907829979=907825303
>>
>> At the time, there was no stand-alone Cap'n Proto article. It's possible
>> the editor decided that anything not notable enough to have its own article
>> should be removed from the table.
>>
>> Since that time, it looks like someone added an article:
>>
>> https://en.wikipedia.org/wiki/Cap%27n_Proto
>>
>> So maybe it makes sense to restore the row in the table now?
>>
>> I suspect Wikipedia rules prohibit me from doing this personally.
>>
>> -Kenton
>>
>> On Thu, Sep 29, 2022 at 1:45 PM Niels Andersen 
>> wrote:
>>
>>> It would be great if someone could add Cap'n Proto to this page
>>> https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats
>>> - I don't have enough knowledge to do it.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/0480d668-00b7-4503-b9fd-836946f8035en%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/135238ba-8d5e-4cb7-bbc8-c2fc4a5a86e4n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmK-LeDaopH7UkQpoANO%3DAXuexMnxxiD56AvEst0aG7pQ%40mail.gmail.com.

Re: [capnproto] Best practice for dynamic_cast equivalent method for extended interfaces

['Kenton Varda' via Cap'n Proto]

Hi Christophe,

The problem is here:

On Mon, Oct 10, 2022 at 5:41 AM Christophe Alexandre <
christophe.alexan...@zellij.io> wrote:

> auto objectInfo = promise.getObjectInfo();
>
auto object = objectInfo.getObject();
>

Note that you are operating on a promise here. The RPC has not actually
completed yet. The `object` you have obtained here is a promised future
object. You can start calling methods on it before the original call has
actually completed. This is called "promise pipeline" -- or, jokingly,
"time travel", on the web site. https://capnproto.org/rpc.html


> bool isA = objectInfo.getIsA(); //not implemented
>

This part doesn't work because pipelining on a boolean value is not
supported. Instead, you need to wait for the promise to complete, and then
you can access the boolean value on the result:

auto promise2 = promise.then([](auto response) {
  bool isA = response.getObjectInfo().getIsA();
});

Or, if you have a WaitScope:

auto response = promise.wait(waitScope);
bool isA = response.getObjectInfo().getIsA();

-Kenton


>
> Is it because of the Bool type ?
>
> Thanks !
> Christophe
>
> Le samedi 8 octobre 2022 à 15:26:42 UTC+2, Christophe Alexandre a écrit :
>
>> Thanks a lot for your very clear answer.
>>
>> Christophe
>>
>> Le vendredi 7 octobre 2022 à 22:42:04 UTC+2, ken...@cloudflare.com a
>> écrit :
>>
>>> Hi Christophe,
>>>
>>> There is no built-in mechanism for this. Instead, you have to build this
>>> into your interface.
>>>
>>> Note that a `Client` object can be "safely" cast to any type using
>>> `client.castAs()`. I say "safely" meaning: you won't
>>> violate C++ memory safety by this cast, even if the server doesn't actually
>>> implement the type. Instead, if you cast to the wrong type, method calls on
>>> that type will fail with UNIMPLEMENTED exceptions. The exception is
>>> produced on the server side; the client side does not actually know the
>>> destination type so cannot predict whether the method is supported.
>>>
>>> To that end, one way to query the type would be to simply try to call a
>>> method on `ConcreteA` or `ConcreteB` and see if it fails with
>>> `exception.getType() == kj::Exception::Type::UNIMPLEMENTED`.
>>>
>>> But if you'd like to avoid the round trip, then I would suggest that
>>> `getAbstracts()` should return a list that contains not just the interface
>>> pointers, but also associated metadata identifying what type it is.
>>>
>>> -Kenton
>>>
>>> On Fri, Oct 7, 2022 at 5:05 AM Christophe Alexandre <
>>> christophe...@zellij.io> wrote:
>>>
 Hi,
 Sorry for the newbie question.

 I've read the following thread and this is related to the same kind of
 question.
 https://groups.google.com/g/capnproto/c/XLo5RPLpVBg/m/LI_sGi72AgAJ

 With following schema example:
 interface Abstract {}
 interface ConcreteA extends(Abstract) {}
 interface ConcreteB extends(Abstract) {}
 interface Object {
   getAbstracts @0 () -> (abstracts :List(Abstract));
 }

 When implementing getAbstracts on the server side, only ConcreteA or
 ConcreteB objects are constructed and returned.
 My question is: on the client side, what is the best practice for
 determining the Abstract object concrete type and casting it to ConcreteA
 or ConcreteB. Something equivalent to C++ dynamic_cast.

 Thanks !
 Christophe Alexandre

 --
 You received this message because you are subscribed to the Google
 Groups "Cap'n Proto" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to capnproto+...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/capnproto/1613eaaf-6025-4eb0-90c3-a7f4a98a2e21n%40googlegroups.com
 
 .

>>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/76ba9ec6-578f-443f-8ef3-75e66ab3e86an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DV-4mDXr9P5f-WHZHsQ-F7u19QSJTSdTsZW_CYuzu0xA%40mail.gmail.com.

Re: [capnproto] Best practice for dynamic_cast equivalent method for extended interfaces

['Kenton Varda' via Cap'n Proto]

Hi Christophe,

There is no built-in mechanism for this. Instead, you have to build this
into your interface.

Note that a `Client` object can be "safely" cast to any type using
`client.castAs()`. I say "safely" meaning: you won't
violate C++ memory safety by this cast, even if the server doesn't actually
implement the type. Instead, if you cast to the wrong type, method calls on
that type will fail with UNIMPLEMENTED exceptions. The exception is
produced on the server side; the client side does not actually know the
destination type so cannot predict whether the method is supported.

To that end, one way to query the type would be to simply try to call a
method on `ConcreteA` or `ConcreteB` and see if it fails with
`exception.getType() == kj::Exception::Type::UNIMPLEMENTED`.

But if you'd like to avoid the round trip, then I would suggest that
`getAbstracts()` should return a list that contains not just the interface
pointers, but also associated metadata identifying what type it is.

-Kenton

On Fri, Oct 7, 2022 at 5:05 AM Christophe Alexandre <
christophe.alexan...@zellij.io> wrote:

> Hi,
> Sorry for the newbie question.
>
> I've read the following thread and this is related to the same kind of
> question.
> https://groups.google.com/g/capnproto/c/XLo5RPLpVBg/m/LI_sGi72AgAJ
>
> With following schema example:
> interface Abstract {}
> interface ConcreteA extends(Abstract) {}
> interface ConcreteB extends(Abstract) {}
> interface Object {
>   getAbstracts @0 () -> (abstracts :List(Abstract));
> }
>
> When implementing getAbstracts on the server side, only ConcreteA or
> ConcreteB objects are constructed and returned.
> My question is: on the client side, what is the best practice for
> determining the Abstract object concrete type and casting it to ConcreteA
> or ConcreteB. Something equivalent to C++ dynamic_cast.
>
> Thanks !
> Christophe Alexandre
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/1613eaaf-6025-4eb0-90c3-a7f4a98a2e21n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQm%2B7xc5WzGMs1QaBsZBpCENf-Rr74oYW_VoxtEod0%2BnQg%40mail.gmail.com.

Re: [capnproto] Wikipedia "Comparison of data-serialization formats" entry

['Kenton Varda' via Cap'n Proto]

It used to be there, but someone removed it, with the comment "clean list
of less notable formats".

https://en.wikipedia.org/w/index.php?title=Comparison_of_data-serialization_formats=revision=907829979=907825303

At the time, there was no stand-alone Cap'n Proto article. It's possible
the editor decided that anything not notable enough to have its own article
should be removed from the table.

Since that time, it looks like someone added an article:

https://en.wikipedia.org/wiki/Cap%27n_Proto

So maybe it makes sense to restore the row in the table now?

I suspect Wikipedia rules prohibit me from doing this personally.

-Kenton

On Thu, Sep 29, 2022 at 1:45 PM Niels Andersen 
wrote:

> It would be great if someone could add Cap'n Proto to this page
> https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats -
> I don't have enough knowledge to do it.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/0480d668-00b7-4503-b9fd-836946f8035en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQk7j8jPw8q8s%2BiKnkkvj4UQ%3DiAvDs%2B4nk34suoY3jE_YQ%40mail.gmail.com.

Re: [capnproto] Recursive autoReconnect logic

['Kenton Varda' via Cap'n Proto]

Indeed, if you call send(), pull a pipelined capability off of it, and then
drop the Promise that `send()` returned... then the logic to detect
disconnect will never run, because it's chained on the promise that you
canceled.

This does seem like a bug... I think I even remember hitting this before.
One way to work around it is to make sure you save the promise returned by
`send()` off to the side rather than throwing it away, so that it's allowed
to complete the disconnect-detection logic.

It's a bit hard for `ReconnectHook` to do this automatically, because if it
did, then you wouldn't actually be able to cancel calls made through
ReconnectHook... attempts to cancel would be ignored. I guess what you
really want is that the call won't be canceled if a pipelined capability
still exists. I think it would be possible to build that but it might be a
little convoluted...

-Kenton

On Mon, Sep 26, 2022 at 7:31 AM Vaci  wrote:

> I'm trying to add reconnection behaviour to capabilities with
> autoReconnect, but the behaviour isn't what I'd expect when I have multiple
> levels of reconnecting capabilities that are pipelined:
>
> Given a trivial schema that obtains one capability from another:
>
> reconnect-test.capnp:
> @0xa0fdcabdd0230110;
> interface TestB {
>   getA @0 () -> (a : Capability);
> }
>
> This example code deliberately creates capability B as disconnected, so
> I'd expect an attempt to reconnect the B capability, but instead, only
> reconnection of the A capability is attempted:
>
> reconnect-test.cpp:
> #include "reconnect-test.capnp.h"
> #include 
> #include 
> #include 
> #include 
>
> int main(int argc, char* argv[]) {
>   kj::EventLoop loop;
>   kj::WaitScope waitScope{loop};
>
>   auto b = capnp::autoReconnect([&](){
> KJ_LOG(WARNING, "Reconnecting B");
> return TestB::Client{KJ_EXCEPTION(DISCONNECTED)};
>   });
>
>   auto a = capnp::autoReconnect([&]{
> KJ_LOG(WARNING, "Reconnecting A");
> return b.getARequest().send().getA();
>   });
>
>   try {
> a.typelessRequest(0, 0, nullptr).send().wait(waitScope);
>   }
>   catch (kj::Exception& exc) {
> KJ_LOG(ERROR, "1", exc);
>   }
>
>   try {
> a.typelessRequest(0, 0, nullptr).send().wait(waitScope);
>   }
>   catch (kj::Exception& exc) {
> KJ_LOG(ERROR, "2", exc);
>   }
> }
>
> Output:
> reconnect-test.cpp:12: warning: Reconnecting B
> reconnect-test.cpp:17: warning: Reconnecting A
> reconnect-test.cpp:17: warning: Reconnecting A
> reconnect-test.cpp:25: error: 1; exc = reconnect-test.cpp:13: disconnected
> reconnect-test.cpp:17: warning: Reconnecting A
> reconnect-test.cpp:32: error: 2; exc = reconnect-test.cpp:13: disconnected
>
> I'd expect the second attempt to result in a reconnection of the broken
> capability B when the (eventual) call to getARequest() occurs?
>
> If I remove the pipelining, and force the call to getARequest to complete,
> then B reconnects just fine.
>
> Vaci
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/19ef7ff7-d419-4208-ab3d-4765a1fc99efn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D%3DK3heCZLrD-_STJ%2BdMzjH44TZnF5FZ-xinzzFdTO3xw%40mail.gmail.com.

Re: [capnproto] Performance of iterating through List vs. raw Data

['Kenton Varda' via Cap'n Proto]

On Wed, Sep 21, 2022 at 9:42 AM Hui min  wrote:

> I want to ask a related question, and found out this thread.
>
> From what I understand in the discussion, if I have
>
> struct Vector3d {
> x @0 :Float64;
> y @1 :Float64;
> z @2 :Float64;
> }
>
> and then
>
> struct Point3Array {
> data @0 :List(Vector3d);
> }
>
> Will i get a perfectly packed array of Vector3d in memory, since it is
> already 8-byte aligned?
>

Yes, in practice, given this schema, the encoder will produce a perfectly
packed array of floats in little-endian byte order.

But note that if you *assume* this to be the case on the reading end, then
you are technically violating Cap'n Proto parsing rules. A Cap'n Proto
parser would normally be expected to accept data where `Vector3d` has been
extended with new fields (newer version of the schema), or where the
trailing fields aren't present (older version of the schema). In these
cases you might not get a perfectly packed array of triplets. You can, of
course, make a rule that these are not allowed for your specific
application and write your code to reject them. I'm just pointing out that
it's a possible difference from what Cap'n Proto's APIs would normally do.


> A follow up question, would there be any changes in the memory layout, if
> I change the definition of Vector3d to be:
>
> struct Vector3d {
> pt @0 :List(Float64)
> }
>
> and always insert 3 elements to Vector3d? I guess this won't work as
> expected, as the List() is inheriently a variable length entity.
>

As you guessed, this case is different. A `List` field is encoded as a
pointer to an external list which can have a different length for each
Vector3d.

-Kenton


>
>
> On Thursday, 15 August 2019 at 3:25:09 pm UTC+8 Philipp Wissmann wrote:
>
>> Hi Kenton
>>
>> Thanks a lot for this amazingly fast reply and the explanations.
>>
>> We might try the direct pointer approach and it's very useful to know
>> this possibility but for now I think having the data in a Data member seems
>> to be working.
>>
>>
>> Cheers!
>> Philipp
>>
>> On Tuesday, August 13, 2019 at 7:43:08 PM UTC+2, Kenton Varda wrote:
>>
>>> Hi Philipp,
>>>
>>> This is a bit of an unusual case, where I imagine you are working with
>>> bulk vector data forming a 3D mesh or some such, and in order to hand if
>>> off to the graphics card, you really need a direct pointer to the
>>> underlying data and you need it to be in a specific layout.
>>>
>>> Using Data is this case might make sense.
>>>
>>> Alternatively, here's one trick I thought of:
>>>
>>> Say your Vector3f is defined as:
>>>
>>> struct Vector3f { x @0 :Float32; y @1 :Float32; z @2 :Float32; }
>>>
>>> Note that Cap'n Proto pads every struct to an 8-byte boundary, so
>>> there's 4 bytes of padding at the end of this struct. If that doesn't work
>>> for you, then I think you have no choice but to use Data. But if the
>>> padding is OK, then here's a way you can get a direct pointer to the data:
>>>
>>> const kj::byte* getRawPointer(capnp::List::Reader list) {
>>>   if (list.size() == 0) {
>>> return nullptr;
>>>   } else {
>>> capnp::AnyStruct::Reader any(list[0]);
>>> KJ_REQUIRE(any.getPointerSection().size() == 0);
>>> KJ_REQUIRE(any.getDataSection().size() == 16);
>>> return any.getDataSection().begin();
>>>   }
>>> }
>>>
>>> Here, you're getting a direct pointer to the "data section" of the first
>>> struct in the list. Structs in a struct list are always stored
>>> contiguously, so you can extend this out to the size of the list. You do
>>> have to verify that the structs have the expected size, since technically
>>> struct sizes are allowed to change to support schema evolution (in this
>>> case, you'll never be able to add fields to Vector3f, except maybe a `w @3
>>> :Float32` which would use the remaining padding without changing the size).
>>> All structs in a struct list have the same size, so if the first struct
>>> looks good, then the whole list is good.
>>>
>>> Hope that helps.
>>>
>>> -Kenton
>>>
>>> On Tue, Aug 13, 2019 at 9:32 AM  wrote:
>>>
>> Hi

 We used Cap'n Proto to serialize data in a shared memory environment
 and defined some message types as structs containining List, i.e.

 struct Message{
points @0 List(Vector3f)
 };

 Where Vector3f is another struct containing either 3 floats or a
 List(Float32). However, extracting the Vector3f's from the List is not fast
 enough for our use cases as you basically need to use std::copy or
 std::transform on the List. We instead now replaced the definition by

 struct Message{
points @0 Data;
 };

 and just read and copy the array of bytes. Unfortunately, this
 basically gets rid of the nice schema language of Cap'n Proto. So, what's
 the most efficient way to define and read a contiguous  array of the same
 data type?

 Best,
 Philipp

 *This message is confidential and 

Re: [capnproto] [Request] Error fix consult

['Kenton Varda' via Cap'n Proto]

Hi Kukkai,

Unfortunately it's pretty unclear what might have gone wrong here, since
all it shows is a bus error. I haven't seen this before. You might have to
dig in yourself with a debugger to get a stack trace.

-Kenton

On Fri, Sep 16, 2022 at 6:25 PM Kukkai W  wrote:

> Dear All,
>
> I would like to use your capnproto-c version 0.9.1.
> But I run the command "make -j6 check". Error was occurred and it told me
> to report it to you. Please see the error on the attached file.
> Could you suggest how to fix this?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/CAM%3DUv9N%2B1j5H8dCZAUwxQo57vjwqKXMvm5BX1LqKTHWwQGUpZQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn%2BXoiTGWn88B3tyhQFJf7kAb9brOPz0TZzESBff8N%2BnA%40mail.gmail.com.

Re: [capnproto] json serialization and deserilization

['Kenton Varda' via Cap'n Proto]

Hi Pavas,

I guess you mean you'd like for your message to be encoded to a JSON array,
rather than an object? The array would be ordered to match the field order?

That's an interesting idea which you could implement with a custom JSON
encoder, but the current provided JSON encoders do not have built-in
support for this, sorry.

-Kenton

On Sun, Sep 18, 2022 at 3:33 PM pavas goyal  wrote:

> Hi,
> I was looking for a serializer that encodes my "json" message in such in
> way that the encoded message just contains the values for json object. I
> have a well defined schema so I don't need the keys in every encoded
> message. I'm using capnp (python wrapper for capnproto) and it seems that
> the encoded message contains keys (making it quite longer than expected).
> Is there any workaround to remove the keys?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/6baa4586-8a65-4095-96a5-1da45c3b6e37n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmZ69mrGP7HzLsGKCeG8TbsME-0sCSzYMvXfd%3DcF2f8jQ%40mail.gmail.com.

Re: [capnproto] CapnProto - Request to other server in callback

['Kenton Varda' via Cap'n Proto]

Hi Thuy,

Your code doesn't work because it tries to use nested WaitScopes. When a
server-side method is called, the event loop is already running, so you
have to use promise.then(lambda) instead of promise.wait(waitScope).

So instead of this:

auto& waitScope = ezClient2.getWaitScope();
{
  auto request = client2.functionSampleRequest();  //Request to SERVER2
  request.setIn(222);
  auto promise = request.send();

  promise.wait(waitScope);

}

return kj::READY_NOW;

Try:

auto request = client2.functionSampleRequest();  //Request to SERVER2
request.setIn(222);
return request.send()
.then([context](capnp::Response
response) {
  // You can handle the response from server2 here if you want.
  // If there's nothing to do, you can replace `.then()` with
`.ignoreResult()`.
})

-Kenton

On Mon, Sep 12, 2022 at 2:40 AM Thuy Doan  wrote:

> Hi all,
>
> I am a newbie to CapnProto. I'd like to have your help.
>
> I want to request a function of server2 in the callback of server1.
> But I got an exception like below.
> Please help me resolve it.
> Many thanks!
>
> * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS 
> (code=1, address=0x30020)
>   * frame #0: 0x00010035859c 
> libcapnp-rpc-0.10.2.dylib`capnp::VatNetwork capnp::rpc::twoparty::ProvisionId, capnp::rpc::twoparty::RecipientId, 
> capnp::rpc::twoparty::ThirdPartyCapId, 
> capnp::rpc::twoparty::JoinResult>::baseConnect(capnp::AnyStruct::Reader) + 20
> frame #1: 0x000100361a2c 
> libcapnp-rpc-0.10.2.dylib`kj::_::TransformPromiseNode kj::Own, capnp::EzRpcClient::Impl::Impl(kj::StringPtr, 
> unsigned int, capnp::ReaderOptions)::'lambda'(kj::Own&&), 
> kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue&) + 512
> frame #2: 0x0001004d2220 
> libkj-async-0.10.2.dylib`kj::_::RunnableImpl::run()
>  + 32
> frame #3: 0x00010028aaa4 
> libkj-0.10.2.dylib`kj::_::runCatchingExceptions(kj::_::Runnable&) + 40
> frame #4: 0x0001004c7e48 
> libkj-async-0.10.2.dylib`kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue&)
>  + 64
> frame #5: 0x0001004c8684 
> libkj-async-0.10.2.dylib`kj::_::ForkHubBase::fire() + 60
> frame #6: 0x0001004c654c 
> libkj-async-0.10.2.dylib`kj::_::waitImpl(kj::Own&&, 
> kj::_::ExceptionOrValue&, kj::WaitScope&, kj::SourceLocation) + 608
> frame #7: 0x00015198 
> client`kj::Promise 
> >::wait(kj::WaitScope&, kj::SourceLocation) + 120
> frame #8: 0x00014b94 client`main + 344
> frame #9: 0x00010003d08c dyld`start + 520
>
> *Sample source code as below*
>
> *SampleServer1.capnp:*
>
> interface SampleServer1 {
>
> callbackRegister @0 (callback :Callback) -> ();   //to register a callback
>
> interface Callback {
> calbackFunc @0 (in :Int32) -> ();
> }
> }
>
> *SampleServer1::Server Impl:*
>
> class SampleServer1Impl : public SampleServer1::Server
> {
> ::kj::Promise callbackRegister(CallbackRegisterContext context){
> auto cb = context.getParams().getCallback());
>
> auto request = cb.calbackFuncRequest();   //Call callback function
> request.setIn(111);
> auto promise = request.send();
>
> return kj::READY_NOW;
>   }
> }
>
> *SampleServer1::Callback::Server Impl:*
>
> class CallbackImpl : public SampleServer1::Callback::Server
> {
> ::kj::Promise calbackFunc(CalbackFuncContext context){
>
> capnp::EzRpcClient ezClient2("unix:/tmp/capnp-server-2");
> Server2::Client client2 = ezClient2.getMain();
>
> auto& waitScope = ezClient2.getWaitScope();
> {
>   auto request = client2.functionSampleRequest();  //Request to SERVER2
>   request.setIn(222);
>   auto promise = request.send();
>
>   promise.wait(waitScope);
>
> }
>
> return kj::READY_NOW;
>   }
> }
>
> *SampleServer2.capnp:*
>
> interface SampleServer2 {
>
> functionSample @0 (in :Int32) -> ();
> }
>
> *SampleServer2::Server impl*
>
> class SampleServer2 Impl : public SampleServer2::Server
> {
> ::kj::Promise functionSample(FunctionSampleContext context){
> //Do something
> return kj::READY_NOW;
>   }
> }
>
> *Client implement*
>
>   capnp::EzRpcClient ezClient("unix:/tmp/capnp-server-1");
>   Subscriber::Client client = ezClient.getMain();
>   auto& waitScope = ezClient.getWaitScope();
>
>  ::SampleServer1::Callback::Client callback = 
> ::SampleServer1::Callback::Client(kj::heap());
>
>   auto request = client.callbackRegisterRequest();  //Register a callback 
> to Server1
>   request.setCallback(callback);
>   auto promise = request.send();
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> 

Re: [capnproto] Promise destructed while it’s being processed

['Kenton Varda' via Cap'n Proto]

Hi Jens,

A couple options:

- You say you can't put the promise in a TaskSet because you need to be
able to specifically cancel it. Could you instead use a kj::Canceler to
control cancellation? The canceler only wraps the promise, then you could
still put it in a TaskSet.

- As a really awful hack, if you know that you are deleting the promise
that is currently running, you could use `promise.detach()` instead.
`detach()` is usually very bad form because it makes the promise
uncancelable, which usually leads to some violations of lifetime
guarantees. However, if you know you're at the end of the promise chain
anyway -- i.e. as soon as you return from the current function, then the
promise is resolved -- then it may be easy to prove that there's no
lifetime issues.

-Kenton

On Sun, Sep 4, 2022 at 12:29 PM Jens Alfke  wrote:

> I’m having some trouble with a Promise stored in a member variable, where
> the ’then’ block attached to the promise calls code that ends up clearing
> that variable. This then triggers a kj exception that a Promise is being
> destructed while it’s being processed. I understand why this is an error,
> but I can’t figure out how else to clean up that variable.
>
> The situation is like:
> std::optional> _promise;  // This is a data member
> of my class
>
> // In a method of my class:
> _promise = something.then([=] {
> …do the work…
> _promise.clear();   // <— boom
> });
> How can I make sure _promise is cleared after its handler completes?
>
> I need a specific reference to the promise — I can’t just shove it into a
> TaskSet — because I have code in other methods that clears _promise when it
> decides that it’s no longer needed. (Specifically, this is an idle timer,
> so when an action occurs it needs to clear the timer.)
>
> I can think of a few ways to fix this, but they feel like hacks. I get the
> feeling this is a situation that comes up in various circumstances — a
> promise whose resolution has to clear the Promise variable. Is there a more
> established best practice for handling it?
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/8CC19E73-C543-4689-ABA9-5D612D98C026%40mooseyard.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DVa5hAQem2soijngw%3D4dnuYP1LYzgzX2m2oEvhUEjVoA%40mail.gmail.com.

Re: [capnproto] Xcode hits SIGUSR1 signal when I use an Executor

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Two options for writing sync-like code that is actually async:
- KJ supports "Fibers", which allocates a separate stack that runs in the
same thread. When running on that stack, you get a `WaitScope` that you can
use to wait on promises. When you wait, the thread switches back to the
main stack.
- We have some support for C++20 coroutines using co_await/co_return (aka
async/await). However, it's currently based on `-fcoroutines-ts` in Clang;
it may not work with any other compiler, and this flag is going to go away
soon, though we'll likely upgrade to the final version of coroutines before
that happens.

-Kenton

On Wed, Aug 31, 2022 at 12:29 PM Jens Alfke  wrote:

>
>
> > On Aug 30, 2022, at 7:37 PM, 'Kenton Varda' via Cap'n Proto <
> capnproto@googlegroups.com> wrote:
> >
> > Hi Jens,
> >
> > It sounds very obnoxious that your debugger insists on breaking on this
> signal.
>
> I’ve filed a bug report with Apple.
>
> > With that said, you could try compiling with `-DKJ_USE_PIPE_FOR_WAKEUP`,
> which causes it to avoid using signals for this. You need to compile both
> KJ itself and your own code that depends on it with this define; if they
> don't match you may get undefined behavior.
>
> For posterity: since I’m building with CMake, I accomplished this by
> adding `add_compile_definitions(KJ_USE_PIPE_FOR_WAKEUP)` to my top-level
> CMakeLists.txt, above the line `add_subdirectory(vendor/capnproto)`.
>
> Now my code can talk inter-thread RPC over the fake socket! Yay!
>
> But of course there’s always another roadblock. The next one I hit is a
> fatal exception "expected !loop.running; wait() is not allowed from within
> event callbacks.” Apparently code running in an Executor block on the
> target thread is not allowed to call Promise.wait()? This is a bummer, as
> so far I’ve been lazy and written my RPC client code in blocking style.
> Looks like it’s time to fully async-ify it.
>
> —Jens

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnMWk2ZROUhaVzr14YnM9yKghdjA1%3Dy7pN94DkkMkWGiQ%40mail.gmail.com.

Re: [capnproto] Xcode hits SIGUSR1 signal when I use an Executor

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Indeed, KJ uses SIGUSR1 to communicate between threads. It sounds very
obnoxious that your debugger insists on breaking on this signal. The signal
is not a problem, it's a normal part of KJ's operation. With that said, you
could try compiling with `-DKJ_USE_PIPE_FOR_WAKEUP`, which causes it to
avoid using signals for this. You need to compile both KJ itself and your
own code that depends on it with this define; if they don't match you may
get undefined behavior.

`kj::newTwoWayPipe()` (the global function) is a completely in-memory
implementation of TwoWayPipe which is tied to a single thread. If you use
`kj::AsyncIoProvider::newTwoWayPipe()` instead, it will create an
implementation backed by a socketpair(). However, unfortunately, the KJ
AsyncIoStream wrapper objects are still tied to the particular thread that
created them. In order to communicate between threads, you will need to
manually create a socketpair(), and then use
kj::LowLevelAsyncIoProvider::wrapSocketFd() to create the AsyncIoStream
wrappers. Each thread would need to use its
own LowLevelAsyncIoProvider object to create the AsyncIoStream for its end
of the pipe. Sorry, there isn't currently a more-convenient way to do this.

-Kenton

On Tue, Aug 30, 2022 at 7:38 PM Jens Alfke  wrote:

> TL;DR: Can the two streams created by kj::newTwoWayPipe() be used on
> different threads? It kind of appears not.
>
> I’ve found a workaround, the LLDB command
> process handle --stop false SIGUSR1
> Unfortunately adding it to my .lldbrc file does nothing; I have to enter
> it by hand every time I start the process.
>
> The next roadblock is that my unit tests create a client and a server
> object, then connect them together by calling kj::newTwoWayPipe() and
> giving one end of the pipe to each. This worked fine in a single thread.
> However, now one end (an AsyncIoStream) gets passed into the new background
> thread where the client object lives. I get an exception
> "expected threadLocalEventLoop ==  || threadLocalEventLoop ==
> nullptr; Event armed from different thread than it was created in.  You
> must use Executor to queue events cross-thread.”
>
> From this and the backtrace it looks as though when I write to this end of
> the pipe, it wants to directly notify the other end, which won’t work
> because it’s the wrong thread for that. I was hoping that the streams would
> use normal Unix I/O, since the comment about the TwoWayPipe class says
> "Typically backed by socketpair() system call”.
>
> So how do I set up streams to do I/O between threads?
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/EA5DCF15-8128-4F65-AE98-708AE1B75D1E%40mooseyard.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnRhZViMA7rYbqZP94iw_EV-szgSLafZSnsZu6%3DKZEsxA%40mail.gmail.com.

Re: [capnproto] Callbacks, cancellation and EOF with ByteStreams

['Kenton Varda' via Cap'n Proto]

Today, without three-party handoff, you could always wrap the promise in a
capability and send that. But, yes, this involves a proxy step that 3PH
won't be able to optimize away later... at least not under the current
design.

-Kenton

On Thu, Aug 25, 2022 at 3:08 AM Vaci  wrote:

> Hi Kenton,
>
> That makes sense, thank you!
>
> A theoretical point perhaps is that one might want to keep the handl alive
> by passing it to another longer-lived VAT, whereas the lifetime of a
> promise is bound by the VAT it's running in.
>
> Vaci
>
> On Wednesday, 24 August 2022 at 18:45:32 UTC+1 ken...@cloudflare.com
> wrote:
>
>> The promise approach is more elegant and efficient. Historically it has
>> sometimes led to complications, though. Particularly tricky is
>> bidirectional streaming.
>>
>> subscribe(downstream :Downstream) -> (upstream :Upstream)
>>
>> The problem here is that all messages sent to `upstream` will normally be
>> held until `subscribe()` itself returns a capability which they can be sent
>> to. So if `subscribe()` doesn't return until the downstream direction is
>> done (or canceled), then the upstream doesn't work.
>>
>> HOWEVER, if you're using the C++ implementation, a solution to this was
>> introduced two or so years ago. You can use `context.setPipeline()` to
>> connect the pipelined capabilities without actually completing the RPC.
>>
>> So, assuming you can rely on `setPipeline()` being available, then I
>> would lean towards the promise approach rather than the handle approach.
>>
>> -Kenton
>>
>> On Wed, Aug 24, 2022 at 11:00 AM Vaci  wrote:
>>
>>> If a capability accepts a callback as a method param, such as for
>>> subscriptions, then it seems there are (at least) two possible ways to
>>> handle cancellation:
>>>
>>> a) return a handle that can be dropped to cancel the subscription
>>>
>>> interface Foo {
>>>   subscribe (callback :Callback) -> (handle :Capability);
>>> }
>>>
>>> or b)
>>>
>>> fulfil the promise returned by the method call at EOF, and drop the
>>> promise to cancel the subscription.
>>>
>>> interface Foo {
>>>   subscribe (callback :Callback);
>>> }
>>>
>>> I'm curious to understand fully the pros and cons of each approach,
>>> especially when the callback is a ByteStream, where it seems natural to
>>> return the pumpTo promise as the result of the subscribe call's
>>> implementation?
>>>
>>> Cheers,
>>> Vaci
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/790ea1e0-65e6-44e1-a97a-93e943dd8b34n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/026806ab-0704-46ec-8b30-5101adb882a9n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D86XfoRDWcwZpVq_P3MPFgLEibC5Qxsdt2Zeot9k67Mw%40mail.gmail.com.

Re: [capnproto] pub/sub zero copy message dispatch

['Kenton Varda' via Cap'n Proto]

Hi Gym,

There's no way to do true zero-copy from one RPC to another. You will have
to copy. And yes, if the outgoing messages have to be built after the
incoming RPC has completed, then you have to make two copies, once into a
temporary in-memory structure.

Note that, if you have a complex structure, using `capnp::clone(reader)` to
create the temporary copy is likely more efficient than using tree of plain
C++ objects, as `capnp::clone` makes sure to do only one allocation where
the entire message is stored together. Also, copying between capnp messages
is generally very efficient compared to converting to a different style of
representation and back.

If your message contains very large Text or Data values, you might be able
to avoid copying those by making use of
`Orhpanage::referenceExternalData()`. However, there's no equivalent that
lets you reference whole message trees.

-Kenton

On Thu, Aug 11, 2022 at 9:03 AM Gyorgy Miru  wrote:

> Hi,
>
> I am implementing a publisher/subscriber RPC interface with a large volume
> of messages.
> The server receives the incoming published messages and dispatches them to
> the subscribers after filtering without modifying the original message
> content. The filtering is done on separate worker threads.
>
> My question is whether this can be done with zero copy? The worst case is
> marshalling the received message struct into a native struct than building
> a struct parameter for each subscriber. That would be 1 + N copies.
>
> I tried to keep the original MyStruct::Reader around (that is received as
> a parameter upon publishing) but my understanding and experience is that it
> only holds a weak reference to the underlying data which is released as
> soon as the original context goes out of scope. See the code below for
> clarity.
>
>
> 
> kj::Promise server::Server::publish(PublishContext context) {
>
> Event::Reader event = context.getParams().getEvent();
> // Is there a way to make Event::Reader-s data outlive this scope?
> // Or a convenient way to make a deep copy?
> eventRouter_.insertEvent(event);
>
> return kj::READY_NOW;
> };
>
> 
>
> The other part of the question is if I manage to keep the reader and its
> data (or I just make a new Event::Builder) can I reuse those to craft
> multiple subscriber dispatch requests. After all, the +N copies is that I
> really want to avoid. If i do something like below, would that work and
> would that result in N copies?
>
>
> 
> Event::Builder/Reader event;
> for (auto client: subscribers) {
>auto req = client.dispatchRequest();
>req.setEvent(event);
>req.send();
> }
>
> 
>
> Thank you for your time and help,
> -Gym
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/6dc55379-205d-4acc-ad95-33aa0f577b90n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnWWxs7hPKZhV5XbH1-9v2SCg_NuP09S1wYVX8G-kZqAQ%40mail.gmail.com.

Re: [capnproto] Installtion Error for capnp on unix

['Kenton Varda' via Cap'n Proto]

Hi Shekhar,

Thanks for the report. I think the problem is that your system has OpenSSL
3.0 installed, and KJ's TLS bindings have some issues with it. There is a
PR open to try to address them but it's not ready yet:

https://github.com/capnproto/capnproto/pull/1434

As long as you aren't using KJ's TLS API then it shouldn't affect you.

-Kenton

On Thu, Aug 4, 2022 at 2:08 AM 'Shekhar Suman Patel' via Cap'n Proto <
capnproto@googlegroups.com> wrote:

> Hi!
> Apologies from my side. I didn't want to waste your time. It was my lack
> of knowledge about how to ask such things.
>
> I am attaching the terminal output as a text file. Kindly have a look at
> it.
>
> Any further information you need please let me know.
> Once again sorry for the inconvenience caused from my side.
>
>
>
> * output.txt
> *
>
> *Shekhar Suman Patel*
>
> Developer
>
> 
>
> Cleveron Mobility AS
>
> shekhar.pa...@clevon.com
>
> +372 5639 8680 <+37256398680>
>
> This letter is confidential and is not subject to disclosure to third
> parties.
>
>
> On Thu, 4 Aug 2022 at 09:58, Shekhar Suman Patel 
> wrote:
>
>> Hi!
>> Apologies from my side. I didn't want to waste your time. It was my lack
>> of knowledge about how to ask such things.
>>
>> I am attaching the terminal output as a text file. Kindly have a look at
>> it.
>>
>> Any further information you need please let me know.
>> Once again sorry for the inconvenience caused from my side.
>>
>> *Shekhar Suman Patel*
>>
>> Developer
>>
>> 
>>
>> Cleveron Mobility AS
>>
>> shekhar.pa...@clevon.com
>>
>> +372 5639 8680 <+37256398680>
>>
>> This letter is confidential and is not subject to disclosure to third
>> parties.
>>
>>
>> On Wed, 3 Aug 2022 at 21:46, Jens Alfke  wrote:
>>
>>> (Also, if I may jump in with a pet peeve of mine: *it’s better to post
>>> logs as text than as screenshots*. Screenshots are not searchable, not
>>> accessible, don’t show more than a few dozen lines, and are of course much
>>> bigger. Attach the entire output as a text file, or at least just
>>> copy-and-paste the relevant text. I speak as someone who has had to try to
>>> diagnose way too many problems submitted as useless screenshots. Thanks.)
>>>
>>> —Jens
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/CAJej193o9GcH99GRt60nmxxAxdVOW8MBFDus9DaJETx2aXxxBA%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DN8aqfmt-L5W1E8pbVKFBxERzgcM_W2X%2BHb9YSi_gQtw%40mail.gmail.com.

Re: [capnproto] Time issue using pycapnp async RPC

['Kenton Varda' via Cap'n Proto]

Apple's POSIX support can be quite buggy, and apparently CLOCK_MONOTONIC
was a late addition, so it's entirely possible there's something wrong with
it that we need to work around in userspace.

However, this is the first I've actually heard this problem being reported
and I'm sure plenty of people have run KJ event loops on Mac, so there must
be a little more to it...

-Kenton

On Wed, Jul 6, 2022 at 3:47 PM Andreas Stenius  wrote:

> Thanks for the reply!
>
> OK, so strange things on my end. Got it, see if I'll be able to
> investigate it further. (I'm on a Mac fwiw)
>
> Darwin xx 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:22 PDT
> 2022; root:xnu-8020.121.3~4/RELEASE_X86_64 x86_64
>
> Den ons 6 juli 2022 kl 22:34 skrev Kenton Varda :
>
>> That's very strange! Presumably the timer in use here is the one in
>> UnixEventPort, but it always calls advanceTo() with a time taken
>> from clock_gettime() with CLOCK_MONOTONIC. That value should never decrease
>> regardless of ntp adjustments. Typically CLOCK_MONOTONIC counts time since
>> system boot.
>>
>> (On Windows, the code is different but should produce the same
>> result: GetTickCount64() and QueryPerformanceCounter() are used to
>> calculate the time. Neither of these should ever go backwards.)
>>
>> So it seems like something weird is going on with your system?
>>
>> -Kenton
>>
>> On Mon, Jul 4, 2022 at 8:05 AM Andreas Stenius  wrote:
>>
>>> Hi all!
>>>
>>> Very long time since I was last here, so happy to see this project is
>>> still active!
>>>
>>> Picked it up for a hobby project to get into it a bit again, this time
>>> using Python.
>>>
>>> I've seen an issue come up quite frequently though, while playing around
>>> with a client/server setup modeled after the asyncio example for pycapnp,
>>> keep hitting this assert
>>> https://github.com/capnproto/capnproto/blob/be7a80b4add706ddaeb41689221146b86c3e0f5f/c%2B%2B/src/kj/timer.c%2B%2B#L113
>>>
>>> libc++abi: terminating with uncaught exception of type
>>> kj::ExceptionImpl: kj/timer.c++:113: failed: expected newTime >= time;
>>> can't advance backwards in time
>>>
>>> stack: 10521bb2e 1052131ee 1050f341c 10507382c 105073672 10507337a
>>> 1033f42b3 1033ec8b4 1033ecae2 1049c12f3 1049c20eb 1033ea4e5 1034e60ff
>>> 10342a701 1033ea833 1034c887f 1033eae9c 1034cbca1 1034c833e 1033eae9c
>>> 1034cbca1 1034c833e 1033eae9c 1034cbca1 1034c833e 1033eae9c 1034cbca1
>>> 1034c833e 1034ccd13 1033eb01c 1034cbca1
>>>
>>> I see this message both on the server and on the client (but with
>>> different stack, naturally). Could this be caused if there's a ntp service
>>> running on my machine that adjusts the time slightly in the "wrong"
>>> direction, perhaps?
>>>
>>> Is this a known issue?
>>>
>>> Cheers,
>>> Andreas kaos Stenius
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/8a94e397-9b55-4b54-aad2-8bd4470b2cc5n%40googlegroups.com
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmkGhAHqFJ8CjviduuRMX9gi%2B6sFQHX%3DhaQJEboFTu%2ByQ%40mail.gmail.com.

Re: [capnproto] Time issue using pycapnp async RPC

['Kenton Varda' via Cap'n Proto]

That's very strange! Presumably the timer in use here is the one in
UnixEventPort, but it always calls advanceTo() with a time taken
from clock_gettime() with CLOCK_MONOTONIC. That value should never decrease
regardless of ntp adjustments. Typically CLOCK_MONOTONIC counts time since
system boot.

(On Windows, the code is different but should produce the same
result: GetTickCount64() and QueryPerformanceCounter() are used to
calculate the time. Neither of these should ever go backwards.)

So it seems like something weird is going on with your system?

-Kenton

On Mon, Jul 4, 2022 at 8:05 AM Andreas Stenius  wrote:

> Hi all!
>
> Very long time since I was last here, so happy to see this project is
> still active!
>
> Picked it up for a hobby project to get into it a bit again, this time
> using Python.
>
> I've seen an issue come up quite frequently though, while playing around
> with a client/server setup modeled after the asyncio example for pycapnp,
> keep hitting this assert
> https://github.com/capnproto/capnproto/blob/be7a80b4add706ddaeb41689221146b86c3e0f5f/c%2B%2B/src/kj/timer.c%2B%2B#L113
>
> libc++abi: terminating with uncaught exception of type kj::ExceptionImpl:
> kj/timer.c++:113: failed: expected newTime >= time; can't advance backwards
> in time
>
> stack: 10521bb2e 1052131ee 1050f341c 10507382c 105073672 10507337a
> 1033f42b3 1033ec8b4 1033ecae2 1049c12f3 1049c20eb 1033ea4e5 1034e60ff
> 10342a701 1033ea833 1034c887f 1033eae9c 1034cbca1 1034c833e 1033eae9c
> 1034cbca1 1034c833e 1033eae9c 1034cbca1 1034c833e 1033eae9c 1034cbca1
> 1034c833e 1034ccd13 1033eb01c 1034cbca1
>
> I see this message both on the server and on the client (but with
> different stack, naturally). Could this be caused if there's a ntp service
> running on my machine that adjusts the time slightly in the "wrong"
> direction, perhaps?
>
> Is this a known issue?
>
> Cheers,
> Andreas kaos Stenius
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/8a94e397-9b55-4b54-aad2-8bd4470b2cc5n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnJE6-fiU%3Dk04nC8SVHBfdQKsxYtOGBZ2qdCN1_KGZyhQ%40mail.gmail.com.

Re: [capnproto] Multiple server threads on same port

['Kenton Varda' via Cap'n Proto]

Hi Peter,

Blocking the event loop may cause a bunch of subtle problems. It may be
possible to get away with if your RPC protocol is incredibly simple
(request/response with one request at a time), but if you do anything more
advanced with Cap'n Proto you'll run into problems. Cap'n Proto was
designed to allow multiple RPCs at the same time, RPCs in both directions,
etc.

If you have to call into a library that only offers a blocking interface, I
would probably recommend calling it from a separate thread. Of course, that
means you need some way to signal between threads, which is a pain. You may
end up needing to use the lower-level KJ async I/O APIs. I would avoid
EzRpcServer which hides too much of the functionality that KJ offers.

It is also possible to use multiple threads. Again this is easiest if you
use the low-level KJ I/O APIs. If you're doing one thread per connection,
it might make sense to just have the thread do an old fashion blocking
accept() call on the listen socket, then run that connection to completion,
then go back and do accept() again. The listen socket itself would be an FD
shared across all threads, so you wouldn't want to use the KJ wrappers for
that.

-Kenton

On Wed, Jun 29, 2022 at 2:33 PM 'Peter Goodman' via Cap'n Proto <
capnproto@googlegroups.com> wrote:

> My current situation is that I have many concurrent clients initiating
> RPCs to a server, and the implementation of these RPCs on the server blocks
> (calls to external APIs). What is the best practice way of handling this
> with Cap'n Proto? Should I run the blocking code on separate threads, and
> have them signal the EzRpcServer server thread somehow? Should I (and can
> I) run multiple EzRpcServer threads, each connecting to the same host:port?
> Or both?
>
> Thank you for any possible recommendations.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/7e39d599-45f9-4208-9c89-6e6996d231een%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQk8ywge9qKzr0mCovA8y5gzEOpyp4a2JE9fNhFmFg2BnQ%40mail.gmail.com.

Re: [capnproto] Fine grained control over event queue

['Kenton Varda' via Cap'n Proto]

Hi Rowan,

Hmm, this feels to me like more of a use case for threads or signal
handlers. Trying to limit processing time by limiting turn count seems
precarious to me as the amount of time spent on any turn could vary wildly.

That said the change you propose seems not too bad and I'd accept it, I
just worry it's the tip of the iceberg here. Like will you find out that
some particular callbacks in the RPC system can be especially expensive,
and need to be counted as multiple turns? I think that would be going to
far.

If by any chance your synchronous processing could be made safe to run in a
signal handler (requires using lockless data structures), I think a timer
signal could be a pretty neat solution here.

-Kenton

On Wed, Jun 22, 2022 at 2:17 PM Rowan Reeve  wrote:

> Hi Kenton,
>
> Our company is attempting to use Cap'n Proto in a soft-realtime system
> where we need tight control over the resolution of promises. Our problem is
> that we have some existing synchronous processing which needs to run
> frequently and timely (on the order of 100 microseconds) but our system
> must also service incoming RPCs (which might access results of said
> processing) without missing a single processing window.
>
> What we would like to be able to do roughly corresponds the below
> pseudocode inside our server:
>
> loop forever:
>   if before synchronous processing window:
> poll single event
>   else:
> do synchronous processing
>
> *WaitScope::poll* almost fulfills our needs, but it causes all new events
> since the last poll to be picked up and their corresponding promises to be
> resolved, in this case to incoming requests, which can occasionally cause
> us to miss our window. Our individual RPC functions are relatively
> short-lived, but when receiving many in a short span of time, we encounter
> this timing issue.
>
> I think what we're looking for is an overload to *WaitScope::poll*
> limiting the turn count:
> https://pastebin.com/T4eFM60F
>
> Your thoughts? Otherwise could this be something that might be accepted as
> a small pull request?
>
> Useful details:
>
>- C++ 17 with official capnp/kj library
>- Linux 5.x kernel (so UnixEventPort as async provider)
>
>
>
> Rowan Reeve
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/17eba638-4bbe-49fe-a270-c5e40b40b967n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnK9rdh7Zv2y0RWT70zhBJkc8WEZeESqRn_Qz1o%2B3RR-g%40mail.gmail.com.

Re: [capnproto] Warnings "EventLoop destroyed with events still in the queue. Memory leak?"

['Kenton Varda' via Cap'n Proto]

Yes, it's normal. I agree it's kind of spammy but I haven't figured out
what to do about it.

On Tue, Jun 7, 2022 at 7:25 PM Jens Alfke  wrote:

>
>
> On Jun 7, 2022, at 3:07 PM, Kenton Varda  wrote:
>
> Info-level logs are generally meant to be enabled only for debugging, so
> the idea here is you normally shouldn't see this log.
>
>
> But I *am* debugging :) I’m debugging my code, using KJ logging. So I do
> see this somewhat scary-looking message mixed in with my own logs.
>
> All exceptions that are transmitted over RPC are info-logged
>
>
> So it's normal for this exception to be created & transmitted when a
> client connection closes? If so I’ll just ignore it from now on.
>
> —Jens
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DVo9QhMAJ_cN1nQ1M-q%2BXZr-tToFooS6krF8Yx%3DLQWiQ%40mail.gmail.com.

Re: [capnproto] Dynamically reading schema const property at runtime (c++)

['Kenton Varda' via Cap'n Proto]

Hi Daniel,

I think you can do:

capnp::ConstSchema type =
schema.getNested("Person").getNested("type").asConst();

But it's been a long time since I touched this.

By the way, please do not use stuff in kj::_ directly, that is a private
namespace. To read a Maybe, use KJ_IF_MAYBE, KJ_ASSERT_NONNULL,
KJ_UNWRAP_OR, etc.

-Kenton

On Wed, Jun 8, 2022 at 12:33 PM Daniel Koohmarey <
daniel.koohma...@braincorp.com> wrote:

> Given the following dummy schema test.capnp:
> struct Person {
>const type :Text = "person.type.example";
>version @0 :UInt32;
>name @1 :Text;
> }
>
> At runtime, how can we read the default const value from a schema struct
> (i.e the "person.type.example" above? (type is not considered a field
> within the struct, but it does appear as a node in the proto):
> std::string schema_filename = "test.capnp"; kj::ArrayPtr kj::StringPtr> import_paths;
> capnp::SchemaParser parser;
> capnp::ParsedSchema schema = parser.parseDiskFile(
> schema_filename.c_str(),
>
>  schema_filename.c_str(), import_paths);
> auto person_maybe = schema.findNested("Person");
> auto* person_schema = kj::_::readMaybe(person_maybe);
>
> // how can I read "person.type.example" out of the Person struct?
>
> Any guidance on the matter is appreciated, thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/389bc2ea-3279-4ddc-b8cc-f554d636767fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DUkTR-6f6LsPMPKhK3QhW9TDcTUR1DcrV6c5zfqSZP_Q%40mail.gmail.com.

Re: [capnproto] Warnings "EventLoop destroyed with events still in the queue. Memory leak?"

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Info-level logs are generally meant to be enabled only for debugging, so
the idea here is you normally shouldn't see this log. All exceptions that
are transmitted over RPC are info-logged but normally you would only want
to log them in production if some actual operation you care about fails
with this exception.

-Kenton

On Tue, Jun 7, 2022 at 5:03 PM Jens Alfke  wrote:

>
>
> On Jun 7, 2022, at 9:53 AM, 'Kenton Varda' via Cap'n Proto <
> capnproto@googlegroups.com> wrote:
>
> Ideally, you should set things up so that when the stack unwinds, all
> Promise objects are destroyed before the EventLoop is destroyed.
>
>
> Thanks! I actually figured this out shortly after posting, but forgot to
> send a follow-up. I had to inspect some classes' member declarations
> carefully to find the two that were in the wrong order; swapping them fixed
> it.
>
> But maybe you can help me with a less-annoying warning that I’m still
> getting. When either the client or the server connection closes, I always
> get an Info-level log:
>
> *returning failure over rpc; exception = capnp/rpc.c++:3107: failed:
> RpcSystem was destroyed.*
>
> It appears harmless, and no exception is actually thrown, but I’d rather
> not be logging this. Any idea what I might still be doing wrong?
>
> —Jens
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQm%3DGZtuX5Jn_QYS1bj2LRJiqtBvFp_som0EkggbYvk0eA%40mail.gmail.com.

Re: [capnproto] Warnings "EventLoop destroyed with events still in the queue. Memory leak?"

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Sorry for the slow reply.

This error indicates that the KJ EventLoop (which is part of the objects
returned by kj::setupAsyncIo()) was destroyed while some kj::Promise
objects still existed.

Ideally, you should set things up so that when the stack unwinds, all
Promise objects are destroyed before the EventLoop is destroyed.
Alternatively, make sure to catch all exceptions at the top level such that
the EventLoop is never destroyed.

-Kenton

On Mon, May 23, 2022 at 6:15 PM Jens Alfke  wrote:

> I’m in the midst of improving the error handling in my code, but there’s a
> warning logged by KJ that I can’t figure out. I’d appreciate any advice or
> insight.
>
> It happens when my client code fails to connect because server-side auth
> fails, so it gets an abrupt "Connection reset by peer” error that’s thrown
> from the first Promise::wait() call. That exception causes my client object
> to get destructed, and during its destructor the warning gets logged (with
> a bunch of backtrace.)
>
> The message as logged from my onRecoverableException handler looks like:
>
> expected head == nullptr [10bf07390 == nullptr]; EventLoop destroyed with
> events still in the queue.  Memory leak?; head->traceEvent() = 10152fc4c
> 101548a28
> kj::_::ImmediatePromiseNode
> >::get(kj::_::ExceptionOrValue&) (in myapp) + 0: returning here
>
> The call stack is pretty long, so I’ve tried to clean it up a bit. “myapp”
> is app code. “snej::shs” is my implementation of the SecretHandshake
> secure-socket protocol, including CapnP glue code adapted from ez-rpc.c++.
>
> myapp::KJExceptionCallback::onRecoverableException() at Logging.cc:80
> :28
> kj::_::Debug::Fault::~Fault() + 172
> kj::EventLoop::~EventLoop() + 432
> kj::_::HeapDisposer kj::Disposer::Dispose_::dispose() at
> memory.h:522:14
> void kj::Disposer::dispose() const at
> memory.h:534:3
> kj::Own::dispose() at memory.h:278:17
> kj::Own::~Own() at memory.h:204:28
> kj::Own::~Own() at memory.h:204:26
> kj::AsyncIoContext::~AsyncIoContext() at async-io.h:876:8
> kj::AsyncIoContext::~AsyncIoContext() at async-io.h:876:8
> snej::shs::SecretRPCContext::~SecretRPCContext() at SecretRPC.cc:56
> :9
> snej::shs::SecretRPCContext::~SecretRPCContext() at SecretRPC.cc:50
> :45
> snej::shs::SecretRPCContext::~SecretRPCContext() at SecretRPC.cc:50
> :45
> kj::Disposer::Dispose_::dispose() at
> memory.h:522:14
> void kj::Disposer::dispose() const at
> memory.h:534:3
> kj::Own::dispose() at memory.h:278:17
> kj::Own::~Own() at memory.h:204:28
> kj::Own::~Own() at memory.h:204:26
> snej::shs::SecretRPCClient::Impl::~Impl() at SecretRPC.cc:283
> :29
> snej::shs::SecretRPCClient::Impl::~Impl() at SecretRPC.cc:283
> :29
> kj::_::HeapDisposer::disposeImpl() const
> at memory.h:433:60
> kj::Disposer::Dispose_::dispose()
> at memory.h:528:14
> void kj::Disposer::dispose() const at
> memory.h:534:3
> kj::Own::dispose() at memory.h:278:17
> kj::Own::~Own() at memory.h:204:28
> kj::Own::~Own() at memory.h:204:26
> snej::shs::SecretRPCClient::~SecretRPCClient() at SecretRPC.cc:383
> :59
> snej::shs::SecretRPCClient::~SecretRPCClient() at SecretRPC.cc:383
> :57
> kj::_::HeapDisposer::disposeImpl() const at
> memory.h:433:60
> kj::Disposer::Dispose_::dispose() at
> memory.h:528:14
> void kj::Disposer::dispose() const at
> memory.h:534:3
> kj::Own::dispose() at memory.h:278:17
> kj::Own::~Own() at memory.h:204:28
> kj::Own::~Own() at memory.h:204:26
> myapp::net::Client::ConnectionImpl::~ConnectionImpl() at Client.cc:281
> :19
> myapp::net::Client::ConnectionImpl::~ConnectionImpl() at Client.cc:281
> :19
> myapp::net::Client::ConnectionImpl::~ConnectionImpl() at Client.cc:281
> :19
> std::__1::default_delete::operator() const at
> unique_ptr.h:57:5
> std::__1::unique_ptr std::__1::default_delete >::reset() at
> unique_ptr.h:318:7
> std::__1::unique_ptr std::__1::default_delete >::~unique_ptr() at
> unique_ptr.h:272:19
> std::__1::unique_ptr std::__1::default_delete >::~unique_ptr() at
> unique_ptr.h:272:17
> myapp::net::Client::~Client() at Client.cc:314 :21
> myapp::net::Client::~Client() at Client.cc:314 :21
> myapp::SyncTask::_run() at SyncTask.cc:166 :5
> myapp::SyncTask::run() const at SyncTask.cc:82 :9
> kj::_::RunnableImpl kj::_::runCatchingExceptions() + 36
> kj::Maybe kj::runCatchingExceptions at exception.h:332:10
> myapp::SyncTask::run() at SyncTask.cc:82 :9
> SyncCommand::runSubcommand() at ClientMain.cc:842
> :24
> InteractiveTool::runSubcommand() at InteractiveTool.cc:72
> :29
> myappClient::run() at ClientMain.cc:178 :13
> 

[capnproto] Re: Cap'n Proto 0.10 release, take 2

['Kenton Varda' via Cap'n Proto]

If you tried to build this on Windows, you may have noticed that it didn't
work, because the file `win32-api-version.h` was missing from the package.

I have now released 0.10.1 to address this.

-Kenton

On Sat, Jun 4, 2022 at 9:45 AM Kenton Varda  wrote:

> Hi Cap'n Proto users,
>
> So, gmail seems to have marked my previous message as malware. The warning
> message made it sound like Gmail thought that the capnproto.org domain
> was malicious, but my theory is that it was actually just upset because I
> linked to a zip file containing an exe. Maybe gmail just always filters
> emails with such links as a precaution. I'm annoyed, though, that the
> warning message made it sound like the domain itself was definitively known
> to serve malware, and that every Google Workspace domain admin who received
> my message at any address at their domain got an alert which claimed
> definitively that the message was malware. AFAICT, this simply isn't true.
> The Google Search Console, VirusTotal, etc. all report the domain is clean.
>
> Assuming my hypothesis about the real problem is correct, perhaps a link
> to the installation page will not be flagged? Let's find out.
>
> Cap'n Proto 0.10 is now available here:
>
> https://capnproto.org/install.html
>
> No big changes, but lots of little fixes.
>
> -Kenton
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkHMvXWTzZNo%3DHfvpx3-Zayg6vb3HneS2J_E4BsW_FhVQ%40mail.gmail.com.

[capnproto] Cap'n Proto 0.10 release, take 2

['Kenton Varda' via Cap'n Proto]

Hi Cap'n Proto users,

So, gmail seems to have marked my previous message as malware. The warning
message made it sound like Gmail thought that the capnproto.org domain was
malicious, but my theory is that it was actually just upset because I
linked to a zip file containing an exe. Maybe gmail just always filters
emails with such links as a precaution. I'm annoyed, though, that the
warning message made it sound like the domain itself was definitively known
to serve malware, and that every Google Workspace domain admin who received
my message at any address at their domain got an alert which claimed
definitively that the message was malware. AFAICT, this simply isn't true.
The Google Search Console, VirusTotal, etc. all report the domain is clean.

Assuming my hypothesis about the real problem is correct, perhaps a link to
the installation page will not be flagged? Let's find out.

Cap'n Proto 0.10 is now available here:

https://capnproto.org/install.html

No big changes, but lots of little fixes.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkq4OpDLZwJeaK2iuxdAPtBPA8H29HZCVd6LTJbzXcoRg%40mail.gmail.com.

[capnproto] Cap'n Proto 0.10 released

['Kenton Varda' via Cap'n Proto]

I've released Cap'n Proto 0.10:

https://capnproto.org/capnproto-c++-0.10.0.tar.gz
https://capnproto.org/capnproto-c++-win32-0.10.0.zip

No big changes, but lots of little fixes.

I wonder if gmail will tag these links as malware, like it did for the
release candidate... >:(

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkwtZY6HzUNLb_41v%2BMtBbExsegMYnmZczreHzASaNbHw%40mail.gmail.com.

[capnproto] Re: v0.10.0 release candidate

['Kenton Varda' via Cap'n Proto]

Oops, probably want to HTTPS those links. (They will redirect, but still.)

https://capnproto.org/capnproto-c++-0.10.0-rc1.tar.gz
https://capnproto.org/capnproto-c++-win32-0.10.0-rc1.zip

-Kenton

On Wed, May 25, 2022 at 8:12 AM Kenton Varda  wrote:

> Hi all,
>
> Just pushing out a release because I haven't in a while. I don't think
> there's any big new stuff in this release, just a few months of incremental
> improvements and maybe some better portability (e.g. auto-detecting whether
> the platform can support fibers).
>
> Here's a candidate to test if you are so inclined:
>
> http://capnproto.org/capnproto-c++-0.10.0-rc1.tar.gz
> http://capnproto.org/capnproto-c++-win32-0.10.0-rc1.zip
>
> -Kenton
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnYkthqCFj5KRKeeM0VgQ%2BJTBgKMQaBPGWCmDsyeuPv6g%40mail.gmail.com.

[capnproto] v0.10.0 release candidate

['Kenton Varda' via Cap'n Proto]

Hi all,

Just pushing out a release because I haven't in a while. I don't think
there's any big new stuff in this release, just a few months of incremental
improvements and maybe some better portability (e.g. auto-detecting whether
the platform can support fibers).

Here's a candidate to test if you are so inclined:

http://capnproto.org/capnproto-c++-0.10.0-rc1.tar.gz
http://capnproto.org/capnproto-c++-win32-0.10.0-rc1.zip

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkDvKgUBJKj0QZ3yyZUs6UBskwHmQu2TbnKXb8uVOE6OQ%40mail.gmail.com.

Re: [capnproto] Building with CMake

['Kenton Varda' via Cap'n Proto]

On Sat, May 14, 2022 at 2:36 PM Jens Alfke  wrote:

>
> On May 13, 2022, at 6:07 PM, Kenton Varda  wrote:
>
> Do you mean that iOS was broken at the last release, but has been fixed at
> master since then? (I don't know how to test iOS so I have no idea which
> versions work or not…)
>
>
> I just mean that AFAIK there are no binary libraries for iOS on your
> website. (Right?)
>

There are no precompiled libraries for any platform. The release package
just contains source code.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnt9MyL1zmQE%2BNZV1mZzo2aOveCZa-yPd-9qFJ0sTL47Q%40mail.gmail.com.

Re: [capnproto] Building with CMake

['Kenton Varda' via Cap'n Proto]

On Fri, May 13, 2022 at 6:31 PM Jens Alfke  wrote:

> I want to build Cap’n Proto from source as part of my project; I know it’s
> preferred to download a release, but I need to support iOS and there are no
> releases for that AFAIK.
>

Do you mean that iOS was broken at the last release, but has been fixed at
master since then? (I don't know how to test iOS so I have no idea which
versions work or not...)


> The build instructions at https://capnproto.org/install.html seem out of
> date; they say to use autoconf, but the current sources have a CMakeFile
> and I found that the usual CMake build command work fine:
>

The instructions are not out of date. autotools is still considered to be
the preferred build system, but cmake files have been contributed. I don't
personally understand cmake so these files are maintained by other
contributors.

(Unfortunately that means I don't know the answer to your question...)

-Kenton


> mkdir build_cmake && cd build_cmake
> cmake ..
> make
>
> However, if I add capnproto as a subdirectory to my project’s
> CMakeLists.txt:
> add_subdirectory(vendor/capnproto)
> it doesn’t build. During the initial ‘cmake’ command for my project, I get
> several errors of the form:
>
> CMake Error at vendor/capnproto/c++/cmake/CapnProtoMacros.cmake:106
> (add_custom_command):
>   Error evaluating generator expression:
>
> $
>
>   No target "CapnProto::capnp_tool"
> Call Stack (most recent call first):
>   CMakeLists.txt:113 (capnp_generate_cpp)
>
>
> I have no idea what this means, or what to do about it…
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/8FDA7AA9-F7A2-4FC7-BC86-DD3B569FF210%40mooseyard.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQke7j9iyCsB8RH%2B2TAcQpCy6L1LAd5HB7qtJxUBc0ETbw%40mail.gmail.com.

Re: [capnproto] Build error on Raspbian, "undefined reference to `__atomic_load_8'"

['Kenton Varda' via Cap'n Proto]

Hi Jens,

I answered on your github issue:
https://github.com/capnproto/capnproto/issues/1448#issuecomment-1093275165

-Kenton

On Fri, Apr 8, 2022 at 2:30 PM Jens Alfke  wrote:

> I'm building CapnP 0.9.1 from source on a new Raspberry Pi 4 running
> Raspbian (because apt only has version 0.7.)
>
> curl -O https://capnproto.org/capnproto-c++-0.9.1.tar.gz
> tar zxf capnproto-c++-0.9.1.tar.gz
> cd capnproto-c++-0.9.1
> ./configure
> make -j6 check
> sudo make install
>
>
> Everything went fine until the final "sudo make install" step:
>
> ...
> libtool: link: g++ -I./src -I./src -DKJ_HEADER_WARNINGS
> -DCAPNP_HEADER_WARNINGS -DCAPNP_INCLUDE_DIR=\"/usr/local/include\" -pthread
> -O2 -DNDEBUG -pthread -DKJ_HAS_ZLIB -pthread -o .libs/capnp
> src/capnp/compiler/module-loader.o
> src/capnp/compiler/capnp.o  ./.libs/libcapnpc.so ./.libs/libcapnp-json.so
> ./.libs/libcapnp.so ./.libs/libkj.so -lpthread -pthread
> /usr/bin/ld: ./.libs/libcapnp.so: undefined reference to `__atomic_load_8'
> /usr/bin/ld: ./.libs/libcapnp.so: undefined reference to `__atomic_store_8’
>
> I’m guessing the missing symbols are CPU intrinsics for 64-bit atomic
> operations. Could this have something to do with the fact that Raspbian is
> still running in 32-bit on a 64-bit CPU?
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/D31A68E1-75DA-404E-99D4-74A02525A562%40mooseyard.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmPgR8Czm%3D14nVF%2Bc_N%2BDv62jkOAB8O179E0xa6%3DN1EPg%40mail.gmail.com.

Re: [capnproto] Writing tests for RPC-based code?

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Sorry for the slow reply.

Usually when I write tests, I don't actually use the RPC system, but rather
have the client call the server directly running in the same thread.
Remember that you can implicitly turn an `Own` into a
`MyInterface::Client`, and then call it like a client:

MyInterface::Client client = kj::heap();

In theory, whether or not you have an actual RPC connection in between
should be transparent.

-Kenton

On Mon, Feb 21, 2022 at 11:47 AM Jens Alfke  wrote:

> Does anyone have advice on writing C++ automated tests for RPC-based code?
> (FWIW, I use the Catch2 unit-test framework, but I don’t restrict it to
> true ‘unit’ tests.) Say I’ve got an interface file, a class `Client` built
> atop classes generated from that interface, and a class `Server` that
> accepts TCP connections from a `Client`. I want to write a test that
> creates a Client and Server, connects them to each other, and then invokes
> some operations on the Client and asserts the correct results.
>
> The first step of course is creating a connection. It looks like the
> simplest supported mechanism is to open a POSIX pipe and then create kj
> AsyncStreams on its two file descriptors. That means I’ll need an
> alternative Client constructor that takes a file descriptor or stream
> instead of a hostname/port…
>
> I’d love to take a look at anyone’s test code that does stuff like this.
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/CED5FAAE-EC6B-47E7-BDE3-58F3C1DD001F%40mooseyard.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DgXqjV4rV6-eX28aZqL%2BOuvGOu5hnc54vsVTq5_-Z%2BGA%40mail.gmail.com.

Re: [capnproto] Can I do better?

['Kenton Varda' via Cap'n Proto]

Hi Jitesh,

Yes, this is possible, but a bit harder.

Instead of using MallocMessageBuilder, you would need to define your own
subclass of capnp::MessageBuilder that returns the buffers you want to use.

You will also need to keep track of framing on your own -- that is, keep
track of the number of segments and their sizes. Normally Cap'n Proto will
write a prefix on the message called the "segment table" which specifies
the length of each segment. But, the size of that table depends on the
number of segments, which normally isn't known in advance, so you wouldn't
know how much space to leave at the start of your message for it. That
said, if you are sure your message will always fit in the first segment,
then the segment table is not that useful anyway.

Also, if you're manually managing segments, then on the receiving end
you'll need to use capnp::SegmentArrayMessageReader, or write a custom
MessageReader subclass.

-Kenton

On Sat, Feb 26, 2022 at 8:09 AM Jitesh Khandelwal 
wrote:

> Thanks Kenton, this is working well. Now the allocation in
> messageToFlatArray is not happening.
>
> Now, is it possible to build the message in the buffer directly ?
> Currently, I'm building in arena and then calling writeMessage to write
> into the buffer.
>
> --
> auto builder = MallocMessageBuilder(*arena*);
> ...
> ...
> auto message = arrayPtr(reinterpret_cast(*buffer*), N);
>
>
> auto aos = kj::ArrayOutputStream(message);
>
>
> capnp::writeMessage(aos, builder);
>
> --
> On Saturday, 26 February 2022 at 07:07:44 UTC+5:30 ken...@cloudflare.com
> wrote:
>
>> Hi Jitesh,
>>
>> Yes, you can avoid the allocation as follows:
>>
>> 1. Construct a kj::ArrayOutputStream that uses your destination buffer as
>> the output.
>> 2. Use capnp::writeMessage() to write the message to that stream.
>>
>> -Kenton
>>
>> On Fri, Feb 25, 2022 at 12:31 PM Jitesh Khandelwal 
>> wrote:
>>
>>> I am trying to implement a middle layer which serialises some data from
>>> a data structure into a buffer, which is owned by and whose contents are
>>> written on the wire by an outer layer.
>>>
>>> Currently, my code looks as follows
>>>
>>> 
>>> constexpr size_t N = 2048;
>>> char arena_buffer[N]{0};
>>> auto arena = arrayPtr(reinterpret_cast(arena_buffer), N /
>>> sizeof(word));
>>>
>>> Encode (some_data_structure, arena, buffer)
>>> {
>>> auto builder = MallocMessageBuilder(arena);
>>> 
>>> // Step 1. set fields from some_data_structure using builder into
>>> arena
>>> 
>>>
>>> // Step 2. convert to flat array (this allocates another array)
>>> auto message = messageToFlatArray(builder);
>>> auto bytes   = message.asBytes();
>>>
>>> // Step 3. copy to destination
>>> memcpy(buffer, bytes.begin(), bytes.size());
>>> }
>>>
>>> 
>>>
>>> My question is, Can I do better?
>>>
>>> My understanding is that the extra allocation in Step 2, is because of
>>> the segment framing protocol ?
>>>
>>> Can I bypass it somehow, as I am sure that the arena is big enough for
>>> my message and there'll be only 1 segment ?
>>>
>>> And if yes, then may be I can use the buffer as the arena, as in build
>>> directly into the output buffer?
>>>
>>>
>>>
>>> Thanks,
>>> Jitesh
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/a6378e3d-fd2e-4009-8cf4-595011170dc9n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/b09290d5-e2d3-4711-b309-9bcdaaade8e5n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: [capnproto] Having trouble with Github CI

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Do you get that same error in both Ubuntu and MacOS actions? It would seem
pretty surprising if both platforms were getting version-skewed
installations.

Does your source code include any checked-in generated code (.capnp.h
files)? If so then the problem is presumably that the code was generated
with the capnp on your machine, which is a different version from what the
action machines use.

-Kenton

On Sat, Feb 26, 2022 at 3:53 PM Jens Alfke  wrote:

> I’m trying to get my (C++) project to build in Github Actions, but getting
> compile errors because apparently there’s a version mismatch between the
> Cap’n Proto headers and tool.
>
> I don’t build Cap’n Proto as part of the project; instead I use a package
> manager to install it. My CMakeLists.txt has a step to compile my interface:
> find_package(CapnProto CONFIG REQUIRED)
> capnp_generate_cpp(myCapnPSources myCapnPHeaders src/interface.capnp)
>
> This works fine in local builds, both macOS and Ubuntu.
>
> In my Github Actions yaml file, I added a build step that uses the
> get-package action:
>   - name: Install Capn Proto
> uses: mstksg/get-package@v1
> with:
>   brew: capnp
>   apt-get: capnproto libcapnp-dev
>
> Unfortunately the C++ compile then fails with the error: "Version mismatch
> between generated code and library headers.  You must use the same version
> of the Cap'n Proto compiler and library.”
>
> I assume this means that on the Github runner the `capnp` tool is from a
> different version than the headers. I don’t know why that would be, when I
> had apt install both packages. (And why would there be a second
> installation of Cap’n Proto at all?)
>
> Has anyone successfully gotten Github Actions to build C++ based Cap’n
> Proto projects?
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/A1E39982-9835-4E45-B9E1-397CC5B286FF%40mooseyard.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQm-XT%2Bsa8Agtgh-32UboA0RSM7%2BfZ9QTB7q3X-U_OLDYA%40mail.gmail.com.

Re: [capnproto] Can I do better?

['Kenton Varda' via Cap'n Proto]

Hi Jitesh,

Yes, you can avoid the allocation as follows:

1. Construct a kj::ArrayOutputStream that uses your destination buffer as
the output.
2. Use capnp::writeMessage() to write the message to that stream.

-Kenton

On Fri, Feb 25, 2022 at 12:31 PM Jitesh Khandelwal 
wrote:

> I am trying to implement a middle layer which serialises some data from a
> data structure into a buffer, which is owned by and whose contents are
> written on the wire by an outer layer.
>
> Currently, my code looks as follows
>
> 
> constexpr size_t N = 2048;
> char arena_buffer[N]{0};
> auto arena = arrayPtr(reinterpret_cast(arena_buffer), N /
> sizeof(word));
>
> Encode (some_data_structure, arena, buffer)
> {
> auto builder = MallocMessageBuilder(arena);
> 
> // Step 1. set fields from some_data_structure using builder into arena
> 
>
> // Step 2. convert to flat array (this allocates another array)
> auto message = messageToFlatArray(builder);
> auto bytes   = message.asBytes();
>
> // Step 3. copy to destination
> memcpy(buffer, bytes.begin(), bytes.size());
> }
>
> 
>
> My question is, Can I do better?
>
> My understanding is that the extra allocation in Step 2, is because of the
> segment framing protocol ?
>
> Can I bypass it somehow, as I am sure that the arena is big enough for my
> message and there'll be only 1 segment ?
>
> And if yes, then may be I can use the buffer as the arena, as in build
> directly into the output buffer?
>
>
>
> Thanks,
> Jitesh
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/a6378e3d-fd2e-4009-8cf4-595011170dc9n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkanKYk%3D1dgdB%2BkDFc7xR%3DQ4o5d%3DoW1o4kBk7RaBu5__g%40mail.gmail.com.

Re: [capnproto] Detect closed connection using StreamFdMessageReader?

['Kenton Varda' via Cap'n Proto]

So, StreamFdMessageReader was designed to read one message from an FD. It
doesn't really have any way to say "try to read a message, but let me know
if it's EOF".

Things you could do instead:
- Use poll() to check for POLLHUP before constructing
StreamFdMessageReader. If POLLHUP is indicated, use ioctl FIONREAD to check
if there are any bytes waiting to be read. If there are not, then you're at
EOF, so don't create a StreamFdMessageReader at all.
- Wrap the FD in an kj::FdInputStream, and wrap that in
kj::BufferedInputStreamWrapper. That lets you call `tryGetReadBuffer()`
which will return null on EOF. `tryGetReadBuffer` doesn't actually advance
the read pointer, it "peeks" at the next bytes in the stream, so if it
returns a non-empty array, you can then use InputStreamMessageReader parse
the message.
- Use KJ's async I/O, in which case you can use capnp::tryReadMessage()
from capnp/serialize-async.h. This attempts to read a message but returns
null on EOF.

Admittedly, these are all work-arounds for a missing API. We really should
have a way to use StreamFdMessageReader but have it tell you if the stream
was EOF...

-Kenton

On Thu, Feb 3, 2022 at 10:47 AM SR D <
software.research.developm...@gmail.com> wrote:

> Hi,
>
> I use capnp::StreamFdMessageReader(fd) to read a message on a file
> descriptor created using network sockets.
>
> And it works fine. However, I'm trying to detect if the client socket has
> closed. Normally, if a read() call returns 0, this would show the client
> connection has closed.
>
> How would I go about this by using StreamFdMessageReader?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/c5b44e93-dde3-41ec-bced-7d839fde6185n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnHv5ak%3D6Fzvf7m2ccAv__TO01_tyKDFp-Fan%3DMerTcXg%40mail.gmail.com.

Re: [capnproto] Capnproto on wasi/emscripten?

['Kenton Varda' via Cap'n Proto]

WASI is essentially a different operating system, POSIX-like but not
exactly the same. We will need to add support for it alongside POSIX and
Windows.

There's actually a PR open to create a build mode which mostly avoids OS
dependencies altogether, which might be a good starting point:

https://github.com/capnproto/capnproto/pull/1376

I'm a little confused about exceptions, though. Exceptions are not an OS
feature, they're a programming language feature... how could they not be
supported? Cap'n Proto actually does support building with
`-fno-exceptions`, so maybe that's a path forward, but there are a lot of
caveats...

-Kenton

On Sat, Jan 22, 2022 at 2:43 PM pepij...@gmail.com 
wrote:

> Hey,
>
> I was playing around a bit to see if I could compile capnproto with the
> wasi-sdk or emscripten. wasi is a "web assembly system interface" that
> allows running wasm code completely without a JS runtime, while emscripten
> requires a browser/nodejs runtime. I think both options could be
> interesting for distributing software in a platform independent way.
>
> However, it seems capnproto doesn't compile out of the box with either.
> When targeting wasi, it is missing a bunch of things such as pipes,
> signals, and exceptions as far as I understand from compile errors in kj
> exception and unix abstraction code. When targeting emscripting it actually
> compiles but has some missing symbols during linking, also related to
> exceptions:
> https://github.com/emscripten-core/emscripten/issues/11985#issuecomment-1019317541
>
> Would there be a way to compile capnproto without these problematic
> features, or a feasible way to provide wasi-specific stubs for them?
>
> Cheers,
> Pepijn
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/52be11e2-9fb9-4a9c-831b-f7798703c5c3n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkzoFnJ82dUCRz07z4AVPUaD3L6ikAAbExaYF%2BaOxNLRA%40mail.gmail.com.

Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

['Kenton Varda' via Cap'n Proto]

No no, this is "block" in the cryptography sense. It's the unit of
computation of the cipher. For salsa20 and chacha20, the block size is 512
bits. You can still start the stream at an arbitrary byte by rounding to
the closest block boundary and then throwing away the extra bytes. (You'd
have to do the XOR'ing manually though, to start at the specific byte you
want.)

But I definitely agree it'd be better to use boxes with authentication.

-Kenton

On Thu, Jan 13, 2022 at 12:19 PM Jens Alfke  wrote:

>
>
> > On Jan 13, 2022, at 9:31 AM, Kenton Varda  wrote:
> >
> > It looks like the chacha20 functions have variants with an "ic"
> parameter, which lets you specify the block counter, but the salsa20
> functions don't have this for some reason.
>
> A block counter would still require dividing the stream into blocks.
> Fixed-size blocks won’t work because the codec will stall until a block is
> completed, which would deadlock most interactive protocols.
> Variable-size blocks depend on the byte counts passed to the writer, which
> then means writing the block size into the output, and assembling a block
> on the read side. This turns out to be just as much work as using the
> higher level APIs like crypto_secretstream_xchacha20poly1305, or for that
> matter crypto_secretbox, both of which authenticate; so might as well just
> use them. (As does Scuttlebutt.)
>
> Bizarrely, there appears to be no actual streaming API where your data
> gets encrypted with successive portions of the infinite cipher stream. This
> is further confirmation of my belief that cryptographers should never be
> allowed to design APIs.
>
> So. Current plan is to write a stream wrapper around crypto_secretbox.
> This involves annoying stuff like buffering data, but it’s not rocket
> science.
>
> —Jens

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DWfGUUyKysUW_ODMt80uRn3XP0Cc%3DzBp%2BexSzocURgbw%40mail.gmail.com.

Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

['Kenton Varda' via Cap'n Proto]

It looks like the chacha20 functions have variants with an "ic" parameter,
which lets you specify the block counter, but the salsa20 functions don't
have this for some reason.

-Kenton

On Wed, Jan 12, 2022 at 6:59 PM Jens Alfke  wrote:

>
> Hmm if you're using a plain xsalsa20 stream and not secret boxes, does
> that mean you're implementing only encryption, not authentication? Note
> that XSalsa20 and related ciphers work by generating a random stream, and
> then XORing it with the plaintext.
>
>
> FYI: It turns out that my stream-encryption code is totally broken anyway.
> I naively believed that Sodium’s `crypto_stream_xor` implemented a stream
> cipher, as the name implies — but it doesn’t. The key and nonce parameters
> are both const, so it’s stateless, and just xor’s the buffer with the same
> bit-stream every time it’s called.
>
> I am not a cryptographer, but I find this baffling and pointless. Why call
> this a “stream cipher” when the API only allows you to encrypt a single
> (variable-size) block of data?
>
> Looks like I’m forced to implement a chunk-based protocol after all. Good
> news is it’ll be tamper-proof.
>
> —Jens
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQncR%3DwDAFW28EXai%3DuYaTTVeS%3DT1W4csLt5UttG0_WkRg%40mail.gmail.com.

Re: [capnproto] Noob question: Can an interface method return null?

['Kenton Varda' via Cap'n Proto]

Any pointer type, including interfaces, can be null.

I think the problem here, though, is that you're making a pipelined call on
the widget before the `wigitNamed()` RPC call has actually completed, and
then later on `widgetNamed()` is completing with a null result, and so the
pipelined call fails. The error you are seeing is a little
confusingly-worded but is in fact the error that would be expected in this
scenario.

So yeah, you *are* returning null correctly, it's just that on the client
side you're trying to make a call to the null capability before the client
actually finds out that it is null.

-Kenton

On Wed, Jan 12, 2022 at 5:05 PM Jens Alfke  wrote:

> This seems like a dumb question, but I’ve been unable to find an answer. *Does
> the RPC system support null values?* In particular, if an RPC method
> returns an interface, is it able to return null?
>
> Say Widget is an interface, and in another interface I’ve got a method
> widgetNamed @1 (name: String) -> (widget: Widget);
> Can this method return null? I’m beginning to suspect it can’t, because
> when it tries to I get an error "Pipeline call on a request that returned
> no capabilities or was already closed”. (The implementation is in C++; to
> return null it just abstains from calling context.getResults().setWidget().)
>
> If it can’t, what is the idiomatic way to provide an optional return
> value? Do I return an exception, or do I create my own `Optional` type as a
> union?
>
> —Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/5862EBA9-F772-44A4-9736-72F70AAB0B86%40mooseyard.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnWLTYVBRgA4saKXSKOEADwo4riEnOqN8DEMSLKoheVLw%40mail.gmail.com.

Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

['Kenton Varda' via Cap'n Proto]

On Wed, Jan 12, 2022 at 12:01 PM Jens Alfke  wrote:

> Yeah, there are no integrity checks in the data stream, and I agree that’s
> a weakness*. Adding MACs requires adding a block- or message-oriented layer
> on top, like SecretBox, the way that Scuttlebutt does. This feels like
> redundant effort since Cap’nP also is itself message-oriented; my guess is
> that there’s a higher level API inside Cap’nP that exposes the message
> framing, and the MAC could be added there, but I have not yet delved deeper
> into the way Cap’nP works. (Hints welcome.)
>

You might want to look at the `capnp::MessageStream` abstraction, instead
of `kj::AsyncIoStream`. It lets you see whole messages, which makes it
easier to customize the framing.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DN_iafGGMxh5kktODf0Bs_gALgO5hFpdmh%3DR94-Ay_3g%40mail.gmail.com.

Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

['Kenton Varda' via Cap'n Proto]

Sorry for the long delay in replying, I had a baby the day you sent this!

This is neat! How many round trips are needed to set up a connection?

When Cap'n Proto gets three-party handoff support, I'm hoping we can do
0-RTT encrypted session setup after introductions. Not many protocols seem
to consider this use case though.

On Tue, Dec 14, 2021 at 2:05 PM Jens Alfke  wrote:

> The handshake also produces two session keys, which are then used to
> encrypt the channel with the 256-bit symmetric XSalsa20 cipher. (This is
> not strictly speaking part of the SecretHandshake protocol, which ends
> after key agreement. Scuttlebutt uses a different encryption scheme based
> on libSodium’s “secret box”.)
>
Hmm if you're using a plain xsalsa20 stream and not secret boxes, does that
mean you're implementing only encryption, not authentication? Note that
XSalsa20 and related ciphers work by generating a random stream, and then
XORing it with the plaintext. So although the attacker can't decrypt the
bytes, they can flip individual bits in the ciphertext and this will result
in the same bit being flipped in the plaintext. Secret boxes add a MAC to
each block which allows the receiver to verify that the bits haven't been
tampered with.

-Kenton

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQn8W%3DyZPQ%3DD%2BjwUAPvQJQj0XR-EKb1yFEtkwpt9jzYCow%40mail.gmail.com.

Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

['Kenton Varda' via Cap'n Proto]

Sounds cool!

Note that the KJ TLS implementation is pretty weird largely to work with
OpenSSL's API being weird. Hopefully the library you're using has a nicer
API and so the implementation will be easier...

-Kenton

On Thu, Dec 9, 2021 at 12:32 PM Jens Alfke  wrote:

> I'm starting to implement support for SecretHandshake
> , a
> "secure-channel based on a a mutually authenticating key agreement
> handshake, with forward secure identity metadata". shs1-c
>  implements the crypto part,
> resulting in a pair of symmetric stream-cipher keys; beyond that I'm going
> to copy and paste and hack the C++ Cap'n Proto TLS code
> ,
> despite being a total newbie at kj.
>
> Basically all I need to do is create a Cap'n Proto RPC connection that
> splices into the TCP I/O and initially does a couple of data exchanges via
> shs1-c, then filters the data streams through the ciphers.
>
> I'm writing in case anyone has knowledge about the kj side of this that
> they'd like to share.
>
> I'll reply here once I've got this working, and I plan to release the code
> as open source.
>
> --Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/fe6b6564-3f08-478e-af5c-2bf461ea0e81n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3D0H-Kkdy_oNWmn7t-65kPMguyRT3JYGz5O09t%3DE8g%2Bmg%40mail.gmail.com.

Re: [capnproto] Unsubscribe in pub/sub model

['Kenton Varda' via Cap'n Proto]

What I usually do is something like:

auto impl = kj::heap();
auto& ref = *impl;
Subscriber::Client client = kj::mv(impl);
rq.setSubscriber(client);

Now you can keep a copy of `client` locally, and as long as it still
exists, then `ref` remains valid -- because `client` is itself a strong
reference to the object.

Note this doesn't work if you need to get notification of when the
`SubscriberImpl` is dropped from the remote side, since holding a strong
ref locally prevents the destructor from being called. If this is an issue,
then you need to do something different. What I usually do here is store a
pointer somewhere, and then have the destructor null out that pointer. This
effectively implements a weak reference.

-Kenton

On Thu, Dec 2, 2021 at 2:22 PM Jens Alfke  wrote:

> I'm also implementing pub-sub, so I was glad to see this thread before I
> wasted too much time. I'm implementing this pattern, but having trouble on
> the client side.
>
> In terms of the example interface, I've created my SubscriberImpl class,
> and written the code to send the "subscribe" message. Now I want to store
> the returned Subscription capability reference in my SubscriberImpl, so it
> can own it and drop it when it's done.
>
> However, I can't figure out how to keep a reference to the SubscriberImpl,
> since I have to move it to the Request object (calling setSubscriber) and
> afterwards it's gone, so I can't call it again.
>
> auto rq = remotePublisher.subscribeRequest();
> auto impl = kj::heap();
> rq.setSubscriber(std::move(impl));
> auto promise = rq.send().then([&](auto response) {return
> response.getSubscription();});
> // *somehow convey the promise to the SubscriberImpl...?*
>
> I'm sure this is just due to my incomplete understanding of how
> Promise/Client/Server objects work...
> Thanks,
>
> --Jens
>
> On Monday, November 22, 2021 at 8:53:42 AM UTC-8 ken...@cloudflare.com
> wrote:
>
>> Hi Mitsuo,
>>
>> I recommend designing the interface like this:
>>
>> interface EventPublisher{
>> interface Subscriber {
>> updateEvent @0 (event: Int32) -> ();
>> }
>>
>> interface Subscription {}
>> subscribe @0 (subscriber: Subscriber) -> (result: Int32,
>> subscription: Subscription);
>> # To unsubscribe, drop the returned `subscription`.
>> }
>>
>>
>> Here, subscribe() returns a `subscription` object. This object has no
>> methods. But, when the capability is dropped, then the destructor will run
>> on the server side. Atn that point, you can remove the subscription.
>>
>> A big advantage of this approach is that it handles connection failures
>> gracefully. If the client randomly disconnects, the `subscription`
>> capability is automatically dropped, thus unsubscribing the client. This
>> way you don't end up stuck sending messages to a disconnected subscriber.
>>
>> It also solves your problem because you can remember arbitrary metadata
>> about the subscriber within the `Subscription` object, so you know what to
>> remove when it's destroyed.
>>
>> -Kenton
>>
>> On Mon, Nov 22, 2021 at 12:40 AM mitsuo 
>> wrote:
>>
>>> Hi,
>>>
>>> I'm trying to implement pub/sub like communication model with the
>>> following scheme.
>>> *pubsub.capnp*
>>> *interface EventPublisher{*
>>> *interface Subscriber {*
>>> *updateEvent @0 (event: Int32) -> ();*
>>> *}*
>>> *subscribe @0 (subscriber: Subscriber) -> (result: Int32);*
>>> *unsubscribe @1 (subscriber: Subscriber) -> (result: Int32);*
>>> *}*
>>>
>>> I'm using *kj::Vector m_subscribers
>>> *to store the current subscribers.
>>> When I try to implement unsubscribe and remove the Subscriber from the
>>> Vector, I couldn't find good method to do that.
>>> Could you give me some advice?
>>>
>>> *server implementation*
>>> *class EventPublisherImpl final : public EventPublisher::Server {*
>>> * protected:*
>>> *  ::kj::Promise subscribe(SubscribeContext context) {*
>>> *cout << "subscribe request received" << endl;*
>>> *m_subscribers.add(context.getParams().getSubscriber());*
>>> *return kj::READY_NOW;*
>>> *  }*
>>>
>>> *  ::kj::Promise unsubscribe(UnsubscribeContext context) {*
>>> *cout << "unsubscribe request received" << endl;*
>>> *auto unsub = context.getParams().getSubscriber();*
>>>
>>> *// I want to remove usub from subscribers like
>>> m_subscribers[unsub].erase();*
>>> *// But I couldn't find a method to compare*
>>> *// "EventPublisher::Subscriber::Client" such as == operator or
>>> public method*
>>> *// to distinguish the client.*
>>> *//*
>>> *// One solution is having an additional argument(id) for this
>>> purpose but*
>>> *// that requres additional management of ID.*
>>> *//  subscribe @0 (subscriber: Subscriber, id: Int32) -> (result:
>>> Int32);*
>>> *//  unsubscribe @1 (id: Int32) -> (result: Int32);*
>>>
>>> *// what I can do is erase everything but this is not my goal*
>>> *

Re: [capnproto] Accept connection in new process

['Kenton Varda' via Cap'n Proto]

Yeah, C++ error messages are the worst. :(

You kind of have to get used to knowing that "can't convert" errors are
usually because you forgot to kj::mv...

-Kenton

On Tue, Nov 30, 2021 at 10:08 AM pepijn de vos 
wrote:

> Ah I solved it... by not storing sim in a variable. Or probably kj::mv
> would have done the job too.
> It's kinda unfortunate the error you get just tells you it's the wrong
> type, rather than.. hey you should move the thing.
> Onwards!!
> Pepijn
>
> On Tue, Nov 30, 2021 at 4:50 PM pepijn de vos 
> wrote:
>
>> Hi Kenton,
>>
>> I got back to this problem and tried implementing the fork idea but ran
>> into another silly problem where the C++ templates are smarter than me it
>> seems.
>> You recommended to not use the EzRPC server, but there is a great lack of
>> examples of other ways to go about it.
>>
>> I found a Rust example that uses the low-level API but of course not
>> exactly the C++ one:
>> https://github.com/capnproto/capnproto-rust/blob/master/capnp-rpc/examples/calculator/server.rs
>> I found some code that's copy-pasted all over using github search that
>> seems to use the lower level API:
>> https://github.com/Nickan/nickan.github.io/blob/b0eb31c33a3b6720606974f9576e663f3b7852ca/drawio/etc/sandstorm/server.c%2B%2B#L420-L422
>>
>> However, I cannot quite get it to work.
>>
>> kj::AsyncIoContext ctx = kj::setupAsyncIo();
>> auto stream = ctx.lowLevelProvider->wrapSocketFd(clientFd);
>> auto network = capnp::TwoPartyVatNetwork(*stream, capnp::rpc::twoparty::
>> Side::CLIENT);
>> kj::Own fs = kj::newDiskFilesystem();
>> const kj::Directory  = fs->getCurrent();
>>
>> auto sim = kj::heap(dir);
>> auto rpc = capnp::makeRpcServer(network, sim);
>>
>> So I set up asyncio, wrap my unix socket FD into a stream, use that to
>> create a network (client or server??)
>> The problem is the call to makeRpcServer. SimulatorImpl is a ::Server
>> subclass.
>> This seems to match the random copy-pasted example, but gives compile
>> errors that it can't convert sim to whatever is expected.
>> I tried reading the source code, which tells me it expects a restorer or
>> a bootstrap, but the restorer is deprecated, but also what EzRPC seems to
>> be using.
>> I tried looking at the tests but that appears to do the same thing, just
>> pass it an owned instance of the implementation??
>>
>> https://github.com/capnproto/capnproto/blob/e5af75ff0513e6d971fa7b48e8108427766e51f9/c%2B%2B/src/capnp/rpc-test.c%2B%2B#L1273-L1279
>> I am at a loss.
>>
>> Pepijn
>>
>>
>> On Thu, May 6, 2021 at 8:22 PM Kenton Varda 
>> wrote:
>>
>>> I don't recommend using EzRPC. That interface turns out to be too
>>> restrictive.
>>>
>>> To set up a KJ event loop, use kj::setupAsyncIo().
>>>
>>> Then to initiate or accept Cap'n Proto RPC connections, use
>>> capnp::TwoPartyClient / capnp::TwoPartyServer.
>>>
>>> For FD passing, create a capnp server object that overrides the virtual
>>> method `kj::Maybe Capability::Server::getFd()` to return an FD number.
>>> Then on the Client objects pointing to that server, you can call `getFd()`
>>> to get that file descriptor -- even from another process, as long as the
>>> connection between the two is a unix domain socket.
>>>
>>> -Kenton
>>>
>>> On Thu, May 6, 2021 at 1:07 PM pepijn de vos 
>>> wrote:
>>>
 Thanks for the suggestions.
 This is still for the simulation server, so each connection is pretty
 expensive anyway, and it's unlikely there will be more in flight than the
 machine has cores available. It might even make sense to have a pool as you
 suggest, and just reject connections if there are no cores available.
 Concern is that you need to supervise these pools in case they crash. With
 fork you can just launch a new process. Any pointers to useful functions
 for this kinds of hacks would be appreciated. I don't have a clue how I'd
 go about obtaining and sending file descriptors over RPC and making new
 servers out of them. I should probably start by studying EzRPC and
 kj::LowLevelAsyncIoProvider.

 Pepijn

 On Thu, May 6, 2021 at 7:48 PM Kenton Varda 
 wrote:

> Ohhh, sorry, I misread. If you haven't started the KJ event loop until
> after the fork, you are probably OK.
>
> You can definitely construct a KJ socket from a raw socket using
> kj::LowLevelAsyncIoProvider.
>
> But if you're OK with the entire connection being handled in the
> isolated process, there are some other options too:
> - Have a pool of processes that are all waiting to accept from the
> same file descriptor. Each connection will only be accepted by one of the
> processes. Make sure that process doesn't accept a new connection until 
> the
> old connection is done.
> - Cap'n Proto supports sending file descriptors between processes in
> RPC messages. So you could have one process that is accepting connections,
> but each time it does, it sends the file descriptor to 

Re: [capnproto] Can't put C++ Client object in std::map

['Kenton Varda' via Cap'n Proto]

This is arguably a bug in the C++ standard library, but the reason it hits
KJ particularly hard is because of KJ's philosophy about constness:

https://github.com/capnproto/capnproto/blob/master/style-guide.md#constness

In particular:
- Constness should be transitive. This implies that copying from a const
value can only be allowed when the copy is a deep copy. Otherwise, making a
copy would discard the transitive constness on the shared backing objects.
Cap'n Proto `Client` objects are copy-by-refcount, so shallow copies, hence
cannot be const copies.
- Constness should imply thread safety. Cap.'n Proto `Client` objects are
refcounted, and the refcounting is not thread-safe (since thread-safe
refcounting is very slow and these objects are tied to a thread-local event
loop anyway).

Unfortunately, the C++ standard library mostly takes the view that `const`
is shallow. To be fair, this is consistent with how built-in pointer and
reference types work, but it is also a much less useful way of using const.

Annoyingly, the C++ standard library containers work just fine with
move-only types, but choke on types that are non-const-copyable. These
containers really ought to default to using move semantics, or
automatically fall back to it when the copy constructor is non-const. I
consider it a bug in the standard library implementation that it errors
instead.

Luckily, you can work around the problem by wrapping the type in a struct
that only has a move constructor, not a copy constructor, which forces the
library to use move semantics only:

struct Wrapper {
  Wrapper(Wrapper&) = delete;
  Wrapper(const Wrapper&) = delete;
  Wrapper(Wrapper&&) = default;
  T value;
}

Now you can use `std::map>`.

Another alternative is to use `kj::HashMap`, which doesn't suffer these
problems, and is faster than `std::unordered_map`.

-Kenton

On Wed, Dec 1, 2021 at 11:59 AM Jens Alfke  wrote:

> Hi! I'm getting started with capnp, using C++ bindings (with C++17
> compiled by Clang 12.) I'm having trouble storing remote object references,
> i.e. xx::Client objects, in STL collections. For example:
>
> std::map foobars;
> ...
> Foobar::Client c = promise.wait(waitScope).getFoobar();
> foobars.insert({name, c});// Compile error
>
> The error is "*the parameter for this explicitly-defaulted copy
> constructor is const, but a member or base requires it to be non-const*."
>
> I eventually worked out that the error is because *Client's copy
> constructor takes a `Client&` parameter, i.e. requires a mutable reference,
> which breaks the std::pair class* because it needs to be able to copy a
> const reference. And if you can't put a client into a std::pair, you can't
> put it into a std::map or std::unordered_map.
>
> I assume there must be a reason for leaving out the standard `const` in
> the copy constructor's parameter; but I can't work out what it would be. I
> know Client objects are wrappers for a pointer to a ref-counted value, so
> copy-constructing one bumps the value's ref-count, but that value is a
> separate object so it shouldn't matter whether the Client reference is
> const or not. (FYI, I've implemented my own ref-counted objects in C++ so
> I'm pretty familiar with the details...)
>
> Is there a good workaround for storing RPC client objects in STL maps?
> Thanks!
>
> --Jens
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/63af6853-ac43-4c75-a161-a4c939199e93n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQnFdZ-gxkT_kCbA8%3DQcMeD1-pWf89Y-obx3r3ti7g7B6w%40mail.gmail.com.