[capnproto] [ACTION REQUIRED] Cap'n Proto mailing list moving to GitHub Discussions

2023-11-20 Thread 'Kenton Varda' via Cap'n Proto
Hi capnproto mailing list, As you have likely noticed, the Cap'n Proto mailing list has had a spam problem lately. I don't know what changed. We never had a problem for 10+ years, and then suddenly, Google couldn't keep the spammers out anymore. Seeing as how Google Groups appears to be a dead

Re: [capnproto] Trying to understand how to use CapnProto

2023-11-15 Thread 'Kenton Varda' via Cap'n Proto
You basically understand correctly, although what you describe is a little bit closer to the Protobuf model than the Cap'n Proto model. They are very similar, but the difference with capnp is that the generated classes do not create a self-contained copy of the message content, but rather act as

Re: [capnproto] v2 modules and versioning

2023-10-31 Thread 'Kenton Varda' via Cap'n Proto
[Sorry for the long delay in replying -- I recently moved into a new house and have been rather swamped.] Cap'n Proto follows in the Protobuf philosophy of versioning, which is, there are no versions, or alternatively, versions are a continuous spectrum. As long as each incremental change is made

Re: [capnproto] Exiting from EzRpcServer

2023-10-30 Thread 'Kenton Varda' via Cap'n Proto
Hi Richard, In the RPC server sample code, there is a line like this: kj::NEVER_DONE.wait(waitScope); This waits forever, running the event loop. If you don't want the loop to run forever, the trick is to wait on a promise that resolves at some point. For instance, you could create one:

Re: [capnproto] Usage of TwoPartyClient() for client callback mechanism

2023-10-24 Thread 'Kenton Varda' via Cap'n Proto
This is expected. The KJ event loop runs when you call `.wait()`. The event loop checks for network events and delivers callbacks. If you haven't called .wait(), then the event loop has no way to run. KJ does not create any background threads for this purpose; the thread that created the event

Re: [capnproto] Clean Disconnect Procedure

2023-10-02 Thread 'Kenton Varda' via Cap'n Proto
On Fri, Sep 29, 2023 at 3:36 PM Alex wrote: > A. How shall a peer discern between an exceptional connection closure > and a normal one? > > My proposal is to add a new message, "GOODBYE", which allows the system > to signal to its remote peer an intent to close the connection. Right, I get the

Re: [capnproto] Clean Disconnect Procedure

2023-09-29 Thread 'Kenton Varda' via Cap'n Proto
t; > > > private: > > > kj::Own> shutdown; > > > }; > > > > > > Another approach could be to add a shutdown() method to FooContext. > > > > > > On the client side, perhaps it is best to simply allow the > > > rpcSystem to fal

Re: [capnproto] Clean Disconnect Procedure

2023-09-29 Thread 'Kenton Varda' via Cap'n Proto
cSystem to > fall out of scope, at which point the destructors can invoke the > necessary machinery to send the GOODBYE and FIN the TCP stream. > > What do you think? Are these approaches going to lead to a leaky > abstraction? Do you know of an elegant way to design this? > > Regard

Re: [capnproto] Clean Disconnect Procedure

2023-09-27 Thread 'Kenton Varda' via Cap'n Proto
(Happy to accept a PR. The relevant code is in `messageLoop()` and `RpcConnectionState::disconnect()` in `rpc.c++`.) On Wed, Sep 27, 2023 at 2:05 PM Kenton Varda wrote: > Indeed, there isn't really a clean shutdown mechanism right now. I guess > it hasn't come up as a priority because in most

Re: [capnproto] Clean Disconnect Procedure

2023-09-27 Thread 'Kenton Varda' via Cap'n Proto
Indeed, there isn't really a clean shutdown mechanism right now. I guess it hasn't come up as a priority because in most use cases we just haven't really cared if there's a TCP RST triggered under the hood... since we're already killing the connection, we ignore that error anyway. I suppose what

Re: [capnproto] Dynamic schemas

2023-09-18 Thread 'Kenton Varda' via Cap'n Proto
Hi Jonathan, I suppose the first question is: Does your database actually need to understand the dynamic fields at all? Or does it just need to be able to echo them back to the client later on, with only the client actually understanding them? If only the client really needs to know about them,

Re: [capnproto] Using Cap'n Proto for FFI

2023-09-06 Thread 'Kenton Varda' via Cap'n Proto
Hi Jonah, Sorry for the very slow reply. This looks cool! Thanks for taking over maintaining the C implementation -- I've added a link to the web site. -Kenton On Fri, Aug 25, 2023 at 10:32 PM Jonah Beckford wrote: > I have an implementation of FFI (foreign function interface) that uses Cap

Re: [capnproto] Random access clarified

2023-09-05 Thread 'Kenton Varda' via Cap'n Proto
On Tue, Sep 5, 2023 at 6:08 AM Johannes Dröge wrote: > Great, thanks for the clear statement! > > I have some follow-up questions now: > > 1) How is the memory usage patter for the deserialization of such members > when they are accessed (either from disk or in memory). I suppose that the >

Re: [capnproto] Random access clarified

2023-09-04 Thread 'Kenton Varda' via Cap'n Proto
Hi Johannes, Yes, it applies to list indexing. -Kenton On Mon, Sep 4, 2023 at 10:43 AM Johannes Dröge wrote: > Hi there! > > The FAQ states *"*Random access*:* You can read just one field of a > message without parsing the whole thing". *However, does that also apply > to List indexing*? I

Re: [capnproto] Re: NAT-Traversal

2023-08-31 Thread 'Kenton Varda' via Cap'n Proto
When you run `capnp compile -orust`, the `capnp` tool will attempt to execute a binary `capnpc-rust` from your $PATH. This binary seems to be provided by the capnpc crate, but declaring a dependency on the crate probably does not automatically install it into your $PATH. You can either install the

Re: [capnproto] Re: NAT-Traversal

2023-08-31 Thread 'Kenton Varda' via Cap'n Proto
You need to install the Rust capnp plugin. I'm not very familiar with Rust but it looks like this is provided by the crate called `capnpc`. https://github.com/capnproto/capnproto-rust https://docs.rs/capnpc/latest/capnpc/ -Kenton On Thu, Aug 31, 2023 at 6:55 AM Tanguille Grootaert <

Re: [capnproto] Re: NAT-Traversal

2023-08-30 Thread 'Kenton Varda' via Cap'n Proto
Hi Tanguille, Cap'n Proto is agnostic to the underlying byte transport. It can layer on top of any byte stream or datagram stream. In terms of the C++ implementation, you can write a custom implementation of the capnp::MessageStream interface in terms of any transport you'd like. For example, you

Re: [capnproto] Schema Refactoring and Unique IDs

2023-08-03 Thread 'Kenton Varda' via Cap'n Proto
n a use case that simultaneously uses dynamic schemas and is trying to reconcile multiple versions of those schemas. -Kenton > > > Jonathan > > > On Wed, Aug 2, 2023 at 3:37 PM 'Kenton Varda' via Cap'n Proto < > capnproto@googlegroups.com> wrote: > >> Yes, changin

Re: [capnproto] Schema Refactoring and Unique IDs

2023-08-02 Thread 'Kenton Varda' via Cap'n Proto
Yes, changing the file ID will change the IDs of all types declared within, unless they have explicitly declared their own IDs. This only really matters if you are using RPC. Type IDs of interfaces are part of the wire protocol. Type IDs of structs and enums are not really used for anything,

[capnproto] Cap'n Proto 1.0 release candidate

2023-07-21 Thread 'Kenton Varda' via Cap'n Proto
Hello capnproto users, I've created a release candidate for version 1.0 of Cap'n Proto's C++ implementation: https://capnproto.org/capnproto-c++-1.0.0-rc1.tar.gz There's nothing particularly special about this release! Frankly, I should probably have applied the 1.0 designation to version 0.6

Re: [capnproto] Flow Control Implementation

2023-07-13 Thread 'Kenton Varda' via Cap'n Proto
RpcSystem::setFlowLimit() should work for this. Yes, this means the RPC system will stop reading messages from the socket buffer if too many requests are in flight. That will naturally cause TCP backpressure to kick in. However, buffering still could happen on the client side if the client isn't

Re: [capnproto] Common Lisp implementation

2023-07-10 Thread 'Kenton Varda' via Cap'n Proto
Cool! Let me know if you have any questions or anything doesn't make sense. -Kenton On Mon, Jul 10, 2023 at 11:02 AM Wojciech Gac wrote: > Hi there, > > I'm about to attempt a Common Lisp implementation of Cap'n Proto. I was > planning to start with serialization and then, provided it goes

Re: [capnproto] Trying to learn how to use capnproto, but running into what seemed to be dead links

2023-06-25 Thread 'Kenton Varda' via Cap'n Proto
It sounds like you were reading the docs inside the github repo. The docs directory there is used to generate the web site at https://capnproto.org . If you read the web site, instead of the github repo, then all the links should work correctly. -Kenton On Sun, Jun 25, 2023 at 7:57 AM

Re: [capnproto] Memory Mapped Reading

2023-06-21 Thread 'Kenton Varda' via Cap'n Proto
Hi Adrian, The memory usage you are seeing happens whether or not you use mmap, it's just accounted differently. If you read the file using many small read() calls, the operating system will still load all of the pages of the file into memory, and will only remove them from memory when the memory

Re: [capnproto] Memory Mapped Reading

2023-06-21 Thread 'Kenton Varda' via Cap'n Proto
Hi Adrian, How are you measuring memory usage, exactly? When using mmap, measuring memory usage gets a bit complicated. The kernel will load pages of the file into memory when you read then, and then it is free to discard those pages at any time -- because it can always load them again later if

Re: [capnproto] [Feature Request] Putting List() object at last and allow reuse of the MallocMessageBuilder

2023-06-05 Thread 'Kenton Varda' via Cap'n Proto
n-wire > reference representation? Is the current representation effectively part of > the API? > > > Jonathan > > > > On Mon, Jun 5, 2023 at 7:32 AM 'Kenton Varda' via Cap'n Proto < > capnproto@googlegroups.com> wrote: > >> Hi Gokul, >> >> There's curr

Re: [capnproto] [Feature Request] Putting List() object at last and allow reuse of the MallocMessageBuilder

2023-06-05 Thread 'Kenton Varda' via Cap'n Proto
Hi Gokul, There's currently no way to "inline" one struct into another. The fundamental problem you have here is this: Say you have a list containing pointer values, and the list is actually at the end of the message. So, you can resize it without fragmentation. You add one new element. Then you

Re: [capnproto] Rules for evolving union members

2023-05-01 Thread 'Kenton Varda' via Cap'n Proto
Hi Andrew, Yes, you got it right. Annoyingly you will have to keep that unused Void in there, but at least it doesn't take any space on the wire. -Kenton On Mon, May 1, 2023 at 4:35 PM 'Andrew Sun' via Cap'n Proto < capnproto@googlegroups.com> wrote: > Suppose I have a schema that looks like

Re: [capnproto] Re: Correctly revoking capabilities holding onto resources

2023-04-17 Thread 'Kenton Varda' via Cap'n Proto
It seems like what's needed here is a way to register a callback on `kj::Canceler` which is invoked when cancel() is called. Then the MembraneHook can register interest in being notified of cancellation, wherein it can permanently disable itself. Actually, there is a tricky way this can be done

Re: [capnproto] PackedMessageReader segfault for larger buffer sizes

2023-04-06 Thread 'Kenton Varda' via Cap'n Proto
Hi Jonathan, I think the problem is with this function: std::shared_ptr uda_capnp_deserialise(const char* bytes, size_t size) { // ArrayPtr requires non-const ptr, but we are only using this to read from the bytes array kj::ArrayPtr buffer(reinterpret_cast(const_cast(bytes)), size);

Re: [capnproto] Re: Correctly revoking capabilities holding onto resources

2023-04-05 Thread 'Kenton Varda' via Cap'n Proto
Hi Rowan, Right, MembranePolicy today uses exclusiveJoin() to effect an asynchronous revocation. The problem with such asynchronous revocation is that it provides no explicit guarantee of when it's safe to assume revocation has occurred. To use this safely, you probably need to detect when your

Re: [capnproto] Re: CapnProto vs gRPC/Protobuf differences - semantics and use case

2023-03-27 Thread 'Kenton Varda' via Cap'n Proto
Hi Jonathan, On Fri, Mar 17, 2023 at 2:10 PM Jonathan Shapiro wrote: > I missed one under semantics: > >- capn-proto structs are defined as reference (pointer) types, while >protobuf message types appear to be value types. > > Does capn-proto support the case where a single struct is

Re: [capnproto] Re: Correctly revoking capabilities holding onto resources

2023-03-27 Thread 'Kenton Varda' via Cap'n Proto
Hi Rowan, Sorry for the slow reply, my inbox is overloaded as always. Indeed, since the `onRevoked` mechanism is triggered by a promise, the actual revocation and cancellation occurs asynchronously. It's possible that some other promise will be queued in between the point where you resolve the

Re: [capnproto] Ada implementation

2023-02-24 Thread 'Kenton Varda' via Cap'n Proto
Hi Ralf, Nope, I don't think I've heard of anyone working on an Ada implementation before! -Kenton On Tue, Feb 21, 2023 at 6:02 AM Ralf Hubert wrote: > Hi, > > I'm thinking about a Ada implementation for Cap'n Proto. > Do you know if there has been already some work done for Ada? > > BR >

Re: [capnproto] Schema Mismatch Between Client and Server

2023-02-14 Thread 'Kenton Varda' via Cap'n Proto
Hi Matt, Your idea would work initially, but if an @3 field were ever added, it could end up incompatible. Instead, assuming `Internal` is a struct type, you can instead declare `internal` to have type `AnyPointer`: struct Event { union { foo @0 : Foo; bar @1 : Bar; internal @2:

Re: [capnproto] Incomplete compile-time Zig implementation

2023-02-01 Thread 'Kenton Varda' via Cap'n Proto
g code during Zig > compilation, but let's not go there. :) > > Yorhel. > > On 2023-01-30, 'Kenton Varda' via Cap'n Proto wrote: > > Hi Yorhel, > > > > Cool stuff. It's neat that Zig can run code at compile time. > > > > Question, though: a

Re: [capnproto] Incomplete compile-time Zig implementation

2023-01-30 Thread 'Kenton Varda' via Cap'n Proto
(Sorry for the very slow reply, my inbox is a mess.) -Kenton On Mon, Jan 30, 2023 at 6:09 PM Kenton Varda wrote: > Hi Yorhel, > > Cool stuff. It's neat that Zig can run code at compile time. > > Question, though: are you calling the C++ parser library to parse Cap'n > Proto schema language, or

Re: [capnproto] Incomplete compile-time Zig implementation

2023-01-30 Thread 'Kenton Varda' via Cap'n Proto
Hi Yorhel, Cool stuff. It's neat that Zig can run code at compile time. Question, though: are you calling the C++ parser library to parse Cap'n Proto schema language, or did you write a whole new parser for it? If the latter, I'm worried as the algorithm to decide the layout of structs is quite

Re: [capnproto] List of List using DynamicList

2023-01-25 Thread 'Kenton Varda' via Cap'n Proto
On Tue, Jan 24, 2023 at 6:11 PM Carlo Contavalli wrote: > Hello, > > On Tue, Jan 24, 2023 at 1:10 PM Kenton Varda > wrote: > >> You mean SchemaLoader::getType(), right? >> > > Correct. > > >> case schema::Type::LIST: >> return ListSchema::of(getType(proto.getList().getElementType(), scope)); >>

Re: [capnproto] List of List using DynamicList

2023-01-24 Thread 'Kenton Varda' via Cap'n Proto
Hi Carlo, It seems like you've figured out what everything represents and than you need to use SchemaLoader here. I'm not sure I understand the problem you're getting at. On Sat, Jan 21, 2023 at 11:30 AM Carlo Contavalli wrote: > I see I can use SchemaLoader.get() to get a capnp::Type out of a

Re: [capnproto] Byte Order of multi byte values

2023-01-24 Thread 'Kenton Varda' via Cap'n Proto
Hi Stefan, Cap'n Proto automatically takes care of byte ordering. You do not need to worry about this. Generally, all modern serialization libraries will take care of endianness automatically. The only time you need to worry about byte-swapping is if you are reading raw bytes from the wire and

Re: [capnproto] Protocol "disembargo" message

2022-12-27 Thread 'Kenton Varda' via Cap'n Proto
The other one that comes to my mind off the top of my head (and is also quite old) is Waterken: https://waterken.sourceforge.net/ Capability people will often distinguish between CapTP-style and "Ken-style" (from Waterken) capabilities. I don't quite remember what the main differences are,

Re: [capnproto] Protocol "disembargo" message

2022-12-26 Thread 'Kenton Varda' via Cap'n Proto
On Mon, Dec 26, 2022 at 4:42 PM Jens Alfke wrote: > Thanks. I then spent some time trying to figure out why this scenario > occurs — when the local peer received the capability Carol from the remote > one, shouldn’t it have been marked in the protocol as being a peer exported > by the recipient?

Re: [capnproto] Protocol "disembargo" message

2022-12-17 Thread 'Kenton Varda' via Cap'n Proto
Hi Jens, To clarify, when I say "Carol lives in Vat A, i.e. next to Alice", I am saying that Alice and Carol are two objects living in the same process. So a capability pointing to Carol was passed across the network and then back again, over the same connection. In this case, we want to shorten

Re: [capnproto] RPC protocol framing details?

2022-12-16 Thread 'Kenton Varda' via Cap'n Proto
Yes, the messages are simply written to the wire one after another. Cap'n Proto messages in the standard serialization are self-delimiting, so there's no need for a framing protocol. -Kenton On Fri, Dec 16, 2022 at 12:07 PM Jens Alfke wrote: > The RPC docs say: > > > The Cap'n Proto RPC

Re: [capnproto] Build error on QNX

2022-12-09 Thread 'Kenton Varda' via Cap'n Proto
Hi Ze, I haven't heard of anyone porting Cap'n Proto to QNX, but it should be possible. `sigaltstack()` is only used as part of `printStackTraceOnCrash()`, which is a totally optional helper that is very OS-specific. You could start by adding some #if/#else such that this function is empty on

Re: [capnproto] Server process stops accepting connections

2022-12-05 Thread 'Kenton Varda' via Cap'n Proto
the AsyncIoStream wrapping your socket, pass the socket file descriptor to another thread, and construct a new AsyncIoStream around it there. -Kenton On Wed, Nov 9, 2022 at 12:19 PM Jens Alfke wrote: > > On Nov 9, 2022, at 6:47 AM, 'Kenton Varda' via Cap'n Proto < > capnproto@googlegroup

Re: [capnproto] capnp::messageToFlatArray to pre-allocated buffer?

2022-12-05 Thread 'Kenton Varda' via Cap'n Proto
Hi Hui, Sorry for the slow reply, I was on leave. messageToFlatArray() is a convenience function for when you don't mind allocation and copying. If you want more control over serialization, you can use MessageBuilder::getSegmentsForOutput() to get direct pointers to the message's underlying

Re: [capnproto] Re: Cap'n Proto CVE-2022-46149

2022-12-02 Thread 'Kenton Varda' via Cap'n Proto
Thanks! On Fri, Dec 2, 2022 at 12:12 PM tripl...@gmail.com wrote: > This is only relevant if you use the pypi packages. If you compile pycapnp > with system libcapnp you should only have to update your system packages. > Note that, unfortunately, you also need to recompile pycapnp against the

[capnproto] Cap'n Proto CVE-2022-46149

2022-11-30 Thread 'Kenton Varda' via Cap'n Proto
Hi capnproto, We have a security advisory today. Although we believe few applications are actually affected, we recommend patching anyway. You can find the details here: https://github.com/capnproto/capnproto/security/advisories/GHSA-qqff-4vw4-f6hx -Kenton -- You received this message because

Re: [capnproto] Fibers and compiling without exceptions

2022-11-25 Thread 'Kenton Varda' via Cap'n Proto
Hi John, Sorry for the very slow response, I've been on leave. Fibers won't be used unless you explicitly ask for them. However, regardless of fibers, you cannot use a version of Cap'n Proto compiled with exceptions enabled inside of an application compiled with exceptions disabled. Compiling

Re: [capnproto] Server process stops accepting connections

2022-11-09 Thread 'Kenton Varda' via Cap'n Proto
Hi Jens, Have you tested whether your server is able to accept concurrent connections normally? E.g. if you open a connection with telnet or something without sending any bytes, leave that open, and then try to use your server, does it work? What does your listen loop look like? -Kenton On

Re: [capnproto] Text blob too big.

2022-10-31 Thread 'Kenton Varda' via Cap'n Proto
For a single Text or Data blob there is a hard limit of 512MB. You can, however, construct a message which contains multiple blobs, e.g. use `List(Text)`. Such a message can be up to 2^64 bytes. If I were redesigning the encoding from scratch I'd probably allow for bigger individual blobs but

Re: [capnproto] Wikipedia "Comparison of data-serialization formats" entry

2022-10-17 Thread 'Kenton Varda' via Cap'n Proto
(Oops, your emails weren't going through from this address because they were landing in the Google Groups moderation queue. I just went and cleared it out (resulting in these dupes, sorry) and allowlisted this address so it won't be moderated in the future.) -Kenton On Mon, Oct 17, 2022 at 12:08

Re: [capnproto] Wikipedia "Comparison of data-serialization formats" entry

2022-10-17 Thread 'Kenton Varda' via Cap'n Proto
Thanks. FWIW I'd argue that Cap'n Proto should have a "no" under "Supports references?". Although the binary encoding uses pointers, implementations generally forbid the message structure to be anything other than tree-shaped. You're not supposed to ever have multiple pointers pointing at the

Re: [capnproto] Best practice for dynamic_cast equivalent method for extended interfaces

2022-10-14 Thread 'Kenton Varda' via Cap'n Proto
Hi Christophe, The problem is here: On Mon, Oct 10, 2022 at 5:41 AM Christophe Alexandre < christophe.alexan...@zellij.io> wrote: > auto objectInfo = promise.getObjectInfo(); > auto object = objectInfo.getObject(); > Note that you are operating on a promise here. The RPC has not actually

Re: [capnproto] Best practice for dynamic_cast equivalent method for extended interfaces

2022-10-07 Thread 'Kenton Varda' via Cap'n Proto
Hi Christophe, There is no built-in mechanism for this. Instead, you have to build this into your interface. Note that a `Client` object can be "safely" cast to any type using `client.castAs()`. I say "safely" meaning: you won't violate C++ memory safety by this cast, even if the server doesn't

Re: [capnproto] Wikipedia "Comparison of data-serialization formats" entry

2022-09-29 Thread 'Kenton Varda' via Cap'n Proto
It used to be there, but someone removed it, with the comment "clean list of less notable formats". https://en.wikipedia.org/w/index.php?title=Comparison_of_data-serialization_formats=revision=907829979=907825303 At the time, there was no stand-alone Cap'n Proto article. It's possible the editor

Re: [capnproto] Recursive autoReconnect logic

2022-09-28 Thread 'Kenton Varda' via Cap'n Proto
Indeed, if you call send(), pull a pipelined capability off of it, and then drop the Promise that `send()` returned... then the logic to detect disconnect will never run, because it's chained on the promise that you canceled. This does seem like a bug... I think I even remember hitting this

Re: [capnproto] Performance of iterating through List vs. raw Data

2022-09-21 Thread 'Kenton Varda' via Cap'n Proto
On Wed, Sep 21, 2022 at 9:42 AM Hui min wrote: > I want to ask a related question, and found out this thread. > > From what I understand in the discussion, if I have > > struct Vector3d { > x @0 :Float64; > y @1 :Float64; > z @2 :Float64; > } > > and then > > struct Point3Array { > data @0

Re: [capnproto] [Request] Error fix consult

2022-09-18 Thread 'Kenton Varda' via Cap'n Proto
Hi Kukkai, Unfortunately it's pretty unclear what might have gone wrong here, since all it shows is a bus error. I haven't seen this before. You might have to dig in yourself with a debugger to get a stack trace. -Kenton On Fri, Sep 16, 2022 at 6:25 PM Kukkai W wrote: > Dear All, > > I would

Re: [capnproto] json serialization and deserilization

2022-09-18 Thread 'Kenton Varda' via Cap'n Proto
Hi Pavas, I guess you mean you'd like for your message to be encoded to a JSON array, rather than an object? The array would be ordered to match the field order? That's an interesting idea which you could implement with a custom JSON encoder, but the current provided JSON encoders do not have

Re: [capnproto] CapnProto - Request to other server in callback

2022-09-12 Thread 'Kenton Varda' via Cap'n Proto
Hi Thuy, Your code doesn't work because it tries to use nested WaitScopes. When a server-side method is called, the event loop is already running, so you have to use promise.then(lambda) instead of promise.wait(waitScope). So instead of this: auto& waitScope = ezClient2.getWaitScope();

Re: [capnproto] Promise destructed while it’s being processed

2022-09-04 Thread 'Kenton Varda' via Cap'n Proto
Hi Jens, A couple options: - You say you can't put the promise in a TaskSet because you need to be able to specifically cancel it. Could you instead use a kj::Canceler to control cancellation? The canceler only wraps the promise, then you could still put it in a TaskSet. - As a really awful

Re: [capnproto] Xcode hits SIGUSR1 signal when I use an Executor

2022-09-02 Thread 'Kenton Varda' via Cap'n Proto
al version of coroutines before that happens. -Kenton On Wed, Aug 31, 2022 at 12:29 PM Jens Alfke wrote: > > > > On Aug 30, 2022, at 7:37 PM, 'Kenton Varda' via Cap'n Proto < > capnproto@googlegroups.com> wrote: > > > > Hi Jens, > > > > It sounds very obnoxious

Re: [capnproto] Xcode hits SIGUSR1 signal when I use an Executor

2022-08-30 Thread 'Kenton Varda' via Cap'n Proto
Hi Jens, Indeed, KJ uses SIGUSR1 to communicate between threads. It sounds very obnoxious that your debugger insists on breaking on this signal. The signal is not a problem, it's a normal part of KJ's operation. With that said, you could try compiling with `-DKJ_USE_PIPE_FOR_WAKEUP`, which causes

Re: [capnproto] Callbacks, cancellation and EOF with ByteStreams

2022-08-25 Thread 'Kenton Varda' via Cap'n Proto
Today, without three-party handoff, you could always wrap the promise in a capability and send that. But, yes, this involves a proxy step that 3PH won't be able to optimize away later... at least not under the current design. -Kenton On Thu, Aug 25, 2022 at 3:08 AM Vaci wrote: > Hi Kenton, > >

Re: [capnproto] pub/sub zero copy message dispatch

2022-08-12 Thread 'Kenton Varda' via Cap'n Proto
Hi Gym, There's no way to do true zero-copy from one RPC to another. You will have to copy. And yes, if the outgoing messages have to be built after the incoming RPC has completed, then you have to make two copies, once into a temporary in-memory structure. Note that, if you have a complex

Re: [capnproto] Installtion Error for capnp on unix

2022-08-04 Thread 'Kenton Varda' via Cap'n Proto
Hi Shekhar, Thanks for the report. I think the problem is that your system has OpenSSL 3.0 installed, and KJ's TLS bindings have some issues with it. There is a PR open to try to address them but it's not ready yet: https://github.com/capnproto/capnproto/pull/1434 As long as you aren't using

Re: [capnproto] Time issue using pycapnp async RPC

2022-07-06 Thread 'Kenton Varda' via Cap'n Proto
Apple's POSIX support can be quite buggy, and apparently CLOCK_MONOTONIC was a late addition, so it's entirely possible there's something wrong with it that we need to work around in userspace. However, this is the first I've actually heard this problem being reported and I'm sure plenty of

Re: [capnproto] Time issue using pycapnp async RPC

2022-07-06 Thread 'Kenton Varda' via Cap'n Proto
That's very strange! Presumably the timer in use here is the one in UnixEventPort, but it always calls advanceTo() with a time taken from clock_gettime() with CLOCK_MONOTONIC. That value should never decrease regardless of ntp adjustments. Typically CLOCK_MONOTONIC counts time since system boot.

Re: [capnproto] Multiple server threads on same port

2022-06-29 Thread 'Kenton Varda' via Cap'n Proto
Hi Peter, Blocking the event loop may cause a bunch of subtle problems. It may be possible to get away with if your RPC protocol is incredibly simple (request/response with one request at a time), but if you do anything more advanced with Cap'n Proto you'll run into problems. Cap'n Proto was

Re: [capnproto] Fine grained control over event queue

2022-06-22 Thread 'Kenton Varda' via Cap'n Proto
Hi Rowan, Hmm, this feels to me like more of a use case for threads or signal handlers. Trying to limit processing time by limiting turn count seems precarious to me as the amount of time spent on any turn could vary wildly. That said the change you propose seems not too bad and I'd accept it, I

Re: [capnproto] Warnings "EventLoop destroyed with events still in the queue. Memory leak?"

2022-06-13 Thread 'Kenton Varda' via Cap'n Proto
Yes, it's normal. I agree it's kind of spammy but I haven't figured out what to do about it. On Tue, Jun 7, 2022 at 7:25 PM Jens Alfke wrote: > > > On Jun 7, 2022, at 3:07 PM, Kenton Varda wrote: > > Info-level logs are generally meant to be enabled only for debugging, so > the idea here is

Re: [capnproto] Dynamically reading schema const property at runtime (c++)

2022-06-08 Thread 'Kenton Varda' via Cap'n Proto
Hi Daniel, I think you can do: capnp::ConstSchema type = schema.getNested("Person").getNested("type").asConst(); But it's been a long time since I touched this. By the way, please do not use stuff in kj::_ directly, that is a private namespace. To read a Maybe, use KJ_IF_MAYBE,

Re: [capnproto] Warnings "EventLoop destroyed with events still in the queue. Memory leak?"

2022-06-07 Thread 'Kenton Varda' via Cap'n Proto
fails with this exception. -Kenton On Tue, Jun 7, 2022 at 5:03 PM Jens Alfke wrote: > > > On Jun 7, 2022, at 9:53 AM, 'Kenton Varda' via Cap'n Proto < > capnproto@googlegroups.com> wrote: > > Ideally, you should set things up so that when the stack unwinds, all > Pr

Re: [capnproto] Warnings "EventLoop destroyed with events still in the queue. Memory leak?"

2022-06-07 Thread 'Kenton Varda' via Cap'n Proto
Hi Jens, Sorry for the slow reply. This error indicates that the KJ EventLoop (which is part of the objects returned by kj::setupAsyncIo()) was destroyed while some kj::Promise objects still existed. Ideally, you should set things up so that when the stack unwinds, all Promise objects are

[capnproto] Re: Cap'n Proto 0.10 release, take 2

2022-06-07 Thread 'Kenton Varda' via Cap'n Proto
If you tried to build this on Windows, you may have noticed that it didn't work, because the file `win32-api-version.h` was missing from the package. I have now released 0.10.1 to address this. -Kenton On Sat, Jun 4, 2022 at 9:45 AM Kenton Varda wrote: > Hi Cap'n Proto users, > > So, gmail

[capnproto] Cap'n Proto 0.10 release, take 2

2022-06-04 Thread 'Kenton Varda' via Cap'n Proto
Hi Cap'n Proto users, So, gmail seems to have marked my previous message as malware. The warning message made it sound like Gmail thought that the capnproto.org domain was malicious, but my theory is that it was actually just upset because I linked to a zip file containing an exe. Maybe gmail

[capnproto] Cap'n Proto 0.10 released

2022-06-03 Thread 'Kenton Varda' via Cap'n Proto
I've released Cap'n Proto 0.10: https://capnproto.org/capnproto-c++-0.10.0.tar.gz https://capnproto.org/capnproto-c++-win32-0.10.0.zip No big changes, but lots of little fixes. I wonder if gmail will tag these links as malware, like it did for the release candidate... >:( -Kenton -- You

[capnproto] Re: v0.10.0 release candidate

2022-05-25 Thread 'Kenton Varda' via Cap'n Proto
Oops, probably want to HTTPS those links. (They will redirect, but still.) https://capnproto.org/capnproto-c++-0.10.0-rc1.tar.gz https://capnproto.org/capnproto-c++-win32-0.10.0-rc1.zip -Kenton On Wed, May 25, 2022 at 8:12 AM Kenton Varda wrote: > Hi all, > > Just pushing out a release

[capnproto] v0.10.0 release candidate

2022-05-25 Thread 'Kenton Varda' via Cap'n Proto
Hi all, Just pushing out a release because I haven't in a while. I don't think there's any big new stuff in this release, just a few months of incremental improvements and maybe some better portability (e.g. auto-detecting whether the platform can support fibers). Here's a candidate to test if

Re: [capnproto] Building with CMake

2022-05-14 Thread 'Kenton Varda' via Cap'n Proto
On Sat, May 14, 2022 at 2:36 PM Jens Alfke wrote: > > On May 13, 2022, at 6:07 PM, Kenton Varda wrote: > > Do you mean that iOS was broken at the last release, but has been fixed at > master since then? (I don't know how to test iOS so I have no idea which > versions work or not…) > > > I just

Re: [capnproto] Building with CMake

2022-05-13 Thread 'Kenton Varda' via Cap'n Proto
On Fri, May 13, 2022 at 6:31 PM Jens Alfke wrote: > I want to build Cap’n Proto from source as part of my project; I know it’s > preferred to download a release, but I need to support iOS and there are no > releases for that AFAIK. > Do you mean that iOS was broken at the last release, but has

Re: [capnproto] Build error on Raspbian, "undefined reference to `__atomic_load_8'"

2022-04-08 Thread 'Kenton Varda' via Cap'n Proto
Hi Jens, I answered on your github issue: https://github.com/capnproto/capnproto/issues/1448#issuecomment-1093275165 -Kenton On Fri, Apr 8, 2022 at 2:30 PM Jens Alfke wrote: > I'm building CapnP 0.9.1 from source on a new Raspberry Pi 4 running > Raspbian (because apt only has version 0.7.) >

Re: [capnproto] Writing tests for RPC-based code?

2022-02-28 Thread 'Kenton Varda' via Cap'n Proto
Hi Jens, Sorry for the slow reply. Usually when I write tests, I don't actually use the RPC system, but rather have the client call the server directly running in the same thread. Remember that you can implicitly turn an `Own` into a `MyInterface::Client`, and then call it like a client:

Re: [capnproto] Can I do better?

2022-02-28 Thread 'Kenton Varda' via Cap'n Proto
Hi Jitesh, Yes, this is possible, but a bit harder. Instead of using MallocMessageBuilder, you would need to define your own subclass of capnp::MessageBuilder that returns the buffers you want to use. You will also need to keep track of framing on your own -- that is, keep track of the number

Re: [capnproto] Having trouble with Github CI

2022-02-28 Thread 'Kenton Varda' via Cap'n Proto
Hi Jens, Do you get that same error in both Ubuntu and MacOS actions? It would seem pretty surprising if both platforms were getting version-skewed installations. Does your source code include any checked-in generated code (.capnp.h files)? If so then the problem is presumably that the code was

Re: [capnproto] Can I do better?

2022-02-25 Thread 'Kenton Varda' via Cap'n Proto
Hi Jitesh, Yes, you can avoid the allocation as follows: 1. Construct a kj::ArrayOutputStream that uses your destination buffer as the output. 2. Use capnp::writeMessage() to write the message to that stream. -Kenton On Fri, Feb 25, 2022 at 12:31 PM Jitesh Khandelwal wrote: > I am trying to

Re: [capnproto] Detect closed connection using StreamFdMessageReader?

2022-02-03 Thread 'Kenton Varda' via Cap'n Proto
So, StreamFdMessageReader was designed to read one message from an FD. It doesn't really have any way to say "try to read a message, but let me know if it's EOF". Things you could do instead: - Use poll() to check for POLLHUP before constructing StreamFdMessageReader. If POLLHUP is indicated, use

Re: [capnproto] Capnproto on wasi/emscripten?

2022-01-24 Thread 'Kenton Varda' via Cap'n Proto
WASI is essentially a different operating system, POSIX-like but not exactly the same. We will need to add support for it alongside POSIX and Windows. There's actually a PR open to create a build mode which mostly avoids OS dependencies altogether, which might be a good starting point:

Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

2022-01-13 Thread 'Kenton Varda' via Cap'n Proto
No no, this is "block" in the cryptography sense. It's the unit of computation of the cipher. For salsa20 and chacha20, the block size is 512 bits. You can still start the stream at an arbitrary byte by rounding to the closest block boundary and then throwing away the extra bytes. (You'd have to

Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

2022-01-13 Thread 'Kenton Varda' via Cap'n Proto
It looks like the chacha20 functions have variants with an "ic" parameter, which lets you specify the block counter, but the salsa20 functions don't have this for some reason. -Kenton On Wed, Jan 12, 2022 at 6:59 PM Jens Alfke wrote: > > Hmm if you're using a plain xsalsa20 stream and not

Re: [capnproto] Noob question: Can an interface method return null?

2022-01-12 Thread 'Kenton Varda' via Cap'n Proto
Any pointer type, including interfaces, can be null. I think the problem here, though, is that you're making a pipelined call on the widget before the `wigitNamed()` RPC call has actually completed, and then later on `widgetNamed()` is completing with a null result, and so the pipelined call

Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

2022-01-12 Thread 'Kenton Varda' via Cap'n Proto
On Wed, Jan 12, 2022 at 12:01 PM Jens Alfke wrote: > Yeah, there are no integrity checks in the data stream, and I agree that’s > a weakness*. Adding MACs requires adding a block- or message-oriented layer > on top, like SecretBox, the way that Scuttlebutt does. This feels like > redundant

Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

2022-01-12 Thread 'Kenton Varda' via Cap'n Proto
Sorry for the long delay in replying, I had a baby the day you sent this! This is neat! How many round trips are needed to set up a connection? When Cap'n Proto gets three-party handoff support, I'm hoping we can do 0-RTT encrypted session setup after introductions. Not many protocols seem to

Re: [capnproto] Implementing SecretHandshake secure-connection support in C++

2021-12-10 Thread 'Kenton Varda' via Cap'n Proto
Sounds cool! Note that the KJ TLS implementation is pretty weird largely to work with OpenSSL's API being weird. Hopefully the library you're using has a nicer API and so the implementation will be easier... -Kenton On Thu, Dec 9, 2021 at 12:32 PM Jens Alfke wrote: > I'm starting to implement

Re: [capnproto] Unsubscribe in pub/sub model

2021-12-03 Thread 'Kenton Varda' via Cap'n Proto
What I usually do is something like: auto impl = kj::heap(); auto& ref = *impl; Subscriber::Client client = kj::mv(impl); rq.setSubscriber(client); Now you can keep a copy of `client` locally, and as long as it still exists, then `ref` remains valid -- because `client` is itself a strong

Re: [capnproto] Accept connection in new process

2021-12-01 Thread 'Kenton Varda' via Cap'n Proto
Yeah, C++ error messages are the worst. :( You kind of have to get used to knowing that "can't convert" errors are usually because you forgot to kj::mv... -Kenton On Tue, Nov 30, 2021 at 10:08 AM pepijn de vos wrote: > Ah I solved it... by not storing sim in a variable. Or probably kj::mv >

Re: [capnproto] Can't put C++ Client object in std::map

2021-12-01 Thread 'Kenton Varda' via Cap'n Proto
This is arguably a bug in the C++ standard library, but the reason it hits KJ particularly hard is because of KJ's philosophy about constness: https://github.com/capnproto/capnproto/blob/master/style-guide.md#constness In particular: - Constness should be transitive. This implies that copying

  1   2   3   4   5   >