Re: [capnproto] Converting AnyStruct to DynamicStruct

2021-11-30 Thread 'Kenton Varda' via Cap'n Proto
Hi Vaci, `anyReader.as(structSchema)` should do it. -Kenton On Tue, Nov 30, 2021 at 2:54 PM Vaci wrote: > Apologies if there is an obvious answer, but f I have a > capnp::StructSchema, and a capnp::AnyStruct::Reader, then can I create a > capnp::DynamicStruct::Reader from them, and if so,

Re: [capnproto] flow limit related deadlock?

2021-11-24 Thread 'Kenton Varda' via Cap'n Proto
It sounds like what Go is providing today would be equivalent to a KJ API that returns two promises: once which waits for backpressure, and one which waits for the RPC return value. In principle, we could provide a new version of `send()` in C++ which provides this. But if it only recognizes

Re: [capnproto] flow limit related deadlock?

2021-11-23 Thread 'Kenton Varda' via Cap'n Proto
On Tue, Nov 23, 2021 at 5:01 PM Ian Denhardt wrote: > Ok, I think I get it, let me know if I have this right: > > The correct thing to do is to handle congestion/flow control for > multiple calls on each object individually, using something like > the mechanisms provided by the C++

Re: [capnproto] flow limit related deadlock?

2021-11-23 Thread 'Kenton Varda' via Cap'n Proto
On Tue, Nov 23, 2021 at 3:59 PM Ian Denhardt wrote: > What are apps *supposed* to do here? It isn't clear to me where else the > backpressure is supposed to come from? > Apps should cap the number of write()s they have in-flight at once. (`-> stream` helps a lot with this, as it'll

Re: [capnproto] flow limit related deadlock?

2021-11-23 Thread 'Kenton Varda' via Cap'n Proto
On Tue, Nov 23, 2021 at 12:41 PM Ian Denhardt wrote: > Wouldn't releasing it on return allow the caller to cause runaway memory > usage by just never sending the finish? the return entry needs to be kept > around in case calls are pipelined on it, and itself might take up some > space (arguably

Re: [capnproto] flow limit related deadlock?

2021-11-23 Thread 'Kenton Varda' via Cap'n Proto
Hmm, I think the intention was that the flow limit should be released on Return, independent of Finish. But I can totally believe I implemented it wrong. Could we just change it to be based on Return? FWIW by default there is no flow limit, it's only enabled in the Sandstorm supervisor to defend

Re: [capnproto] Unsubscribe in pub/sub model

2021-11-22 Thread 'Kenton Varda' via Cap'n Proto
Hi Mitsuo, I recommend designing the interface like this: interface EventPublisher{ interface Subscriber { updateEvent @0 (event: Int32) -> (); } interface Subscription {} subscribe @0 (subscriber: Subscriber) -> (result: Int32, subscription: Subscription); # To

Re: [capnproto] Cloning DynamicStructs

2021-11-19 Thread 'Kenton Varda' via Cap'n Proto
No, there's no fundamental limitation here. I think we just need an overload of `clone()` for this that accounts for the subtle API differences with `DynamicStruct`, e.g. passing the `Schema` in various places. -Kenton On Thu, Nov 18, 2021 at 6:31 AM Vaci wrote: > Hi, > > Should it be possible

Re: [capnproto] compiling with automake

2021-10-21 Thread 'Kenton Varda' via Cap'n Proto
Once the capnp subproject build is done, you should be able to invoke the `capnp` compiler straight from the subproject location. I think you'd have to write a raw make rule that invokes the compiler. You can see how Cap'n Proto's own build does this here:

Re: [capnproto] compiling with automake

2021-10-21 Thread 'Kenton Varda' via Cap'n Proto
It should work as a subpackage. See: https://www.gnu.org/software/automake/manual/html_node/Subpackages.html -Kenton On Thu, Oct 21, 2021 at 12:45 PM SR D < software.research.developm...@gmail.com> wrote: > I would like to make use of capnproto in an autotools project and > wondering if anyone

Re: [capnproto] client to client communication

2021-10-15 Thread 'Kenton Varda' via Cap'n Proto
Hi Leon, Abstractly, what you describe should be easy with Cap'n Proto. One client can send a capability (interface reference) to the server, and the server can freely send that capability on to some other client. Cap'n Proto will automatically arrange to proxy messages from one client to the

[capnproto] Cap'n Proto 0.9.1 release

2021-09-22 Thread 'Kenton Varda' via Cap'n Proto
Hi folks, I just put up release 0.9.1 which fixes a number of small platform-specific build issues that were present in 0.9.0. If you had any compile issues with 0.9.0, hopefully they are fixed now. Also, FreeBSD and OpenBSD should work! (Thanks @sobomax and @ZhanYF.) -Kenton -- You received

[capnproto] Release candidate 0.9.0-rc1

2021-08-08 Thread 'Kenton Varda' via Cap'n Proto
Hi all, Release candidate out. Please let me know if you spot any problems. (But our CI is pretty thorough these days so it seems unlikely.) https://capnproto.org/capnproto-c++-0.9.0-rc1.tar.gz https://capnproto.org/capnproto-c++-win32-0.9.0-rc1.zip -Kenton -- You received this message

Re: [capnproto] generate make dependencies

2021-08-05 Thread 'Kenton Varda' via Cap'n Proto
I think this could be implemented as a regular old code generator plugin that directly produces the deps files. No need to integrate with the C++ plugin nor output an intermediate representation. Note that `capnp compile` supports passing multiple `-o` flags to run multiple plugins, so you can

Re: [capnproto] Thread pool executor

2021-07-30 Thread 'Kenton Varda' via Cap'n Proto
Hi Alessandro, KJ doesn't currently have any built-in thread pool implementation. The KJ async / Promise framework is centered around single-threaded event loops, so probably isn't what you're looking for. I guess if I were implementing this I'd write a class that creates threads using

Re: [capnproto] Cap'n Proto test messages

2021-07-21 Thread 'Kenton Varda' via Cap'n Proto
Cool stuff! The C++ implementation has test.capnp that should cover everything, but is pretty messy, having evolved slowly together with the C++ implementation. Definitely nice to have a separate test repository that is clean and really meant to be consumed by multiple implementations. -Kenton

Re: [capnproto] Selecting the correct message type?

2021-07-19 Thread 'Kenton Varda' via Cap'n Proto
Sorry, I'm not sure what you mean. `myns::Person` isn't a real type, it's just an empty class that acts as a namespace containing the `Reader` and `Builder` types. `Reader` is your "handle to the data", and its getters are the only way to read the underlying data (the getters read directly from

Re: [capnproto] Selecting the correct message type?

2021-07-17 Thread 'Kenton Varda' via Cap'n Proto
There is no way to find out a message's type if you don't know it already. So, you'll need to arrange to communicate the type yourself. The easiest way is to write an "envelope" message that is a union of all your other messages. struct OuterMessage { union { foo @0 :FooMessage; bar @1

Re: [capnproto] How to align capnproto data to architectures word size?

2021-07-17 Thread 'Kenton Varda' via Cap'n Proto
Ideally, you'd update the implementation of `getBuff()` so that it allocates an aligned buffer. Note that malloc() always returns aligned buffers, so if you can make it so that the pointer returned by getBuff() is one that was allocated directly by malloc(), then it should work. Probably, though,

Re: [capnproto] Use with RTOS-based Embedded Systems

2021-06-09 Thread 'Kenton Varda' via Cap'n Proto
Hi Kenny, Avoiding malloc should be easy. With custom subclasses of MessageReader and MessageBuilder, you can do your memory management any way you want. Cap'n Proto's serialization layer is very good about avoiding memory allocation unless you ask for it. (The RPC layer, on the other hand, does

Re: [capnproto] Use with RTOS-based Embedded Systems

2021-06-09 Thread 'Kenton Varda' via Cap'n Proto
Hi Kenny, The code is fairly modular, so I think it should be possible to pull out a subset that has a reasonably small code footprint. However, this is admittedly not a well-traveled path, so you'll be a bit on your own. The maintainers of the C++ implementation (such as myself) mainly use it on

Re: [capnproto] Waiting with a timeout

2021-06-09 Thread 'Kenton Varda' via Cap'n Proto
Yes, I agree there's some missing overview docs regarding Cap'n Proto RPC. The web page is not very detailed. FWIW, I recently wrote an overview/tour of KJ: https://github.com/capnproto/capnproto/blob/master/kjdoc/tour.md But that doesn't cover Cap'n Proto itself. > If I use the setupAsyncIo

Re: [capnproto] Waiting with a timeout

2021-06-07 Thread 'Kenton Varda' via Cap'n Proto
Hi Björn, Instead of Ez RPC, try using kj::setupAsyncIo() and capnp::TwoPartyClient / capnp::TwoPartyServer. These APIs give you more control. In particular, kj::setupAsyncIo() returns an kj::AsyncIoProvider which, among other things, provides a kj::Timer API which you can use for timeouts. KJ's

Re: [capnproto] Copying existing struct into request params

2021-06-04 Thread 'Kenton Varda' via Cap'n Proto
I see. There isn't a great API for this right now. The problem is, once a struct Builder is already allocated, the struct can't be resized. But if you're trying to copy the content of another struct into it, it's possible that other struct was originally created with a newer version of the schema

Re: [capnproto] Copying existing struct into request params

2021-06-02 Thread 'Kenton Varda' via Cap'n Proto
Hi Vaci, An RPC request is always a struct type, not a list, so I assume what you really mean here is that you want to use the list as a field of the request? E.g. you have: struct Foo {} interface Bar { foo @0 (foos :List(Foo)); } Then you would do: auto req = bar.fooRequest(); auto

Re: [capnproto] PosixPath error when loading schema

2021-06-01 Thread 'Kenton Varda' via Cap'n Proto
Hi Kyle, I think you need to convert `capnp_path` back into a string before passing it to `capnp.load()`. Currently it appears you are passing a `Path` object that you constructed using pathlib, but `capnp.load` doesn't know how to deal with those, it wants a plain string. -Kenton On Tue, Jun

Re: [capnproto] Linker error: struct capnp::_::RawSchema const capnp::schemas

2021-05-10 Thread 'Kenton Varda' via Cap'n Proto
Hi Pratik, When you run the Cap'n Proto code generator, it produces two files: a header (ending in `.capnp.h`) and a source file (ending in `.capnp.c++`). Did you make sure to compile the source file into your project? I'm guessing the missing symbol is from there, since it seems to be

Re: [capnproto] Accept connection in new process

2021-05-06 Thread 'Kenton Varda' via Cap'n Proto
Hi Pepijn, Yeah, forking won't work well. I don't recommend that. One problem is that if you fork() with an event loop running, both the parent and child processes will end up with the same epoll FD and so will receive each other's events, which will likely lead to chaos. So I recommend spawning

Re: [capnproto] Generated files .h and .c++

2021-05-04 Thread 'Kenton Varda' via Cap'n Proto
The tables are used to support the "schema" and "dynamic" APIs (aka "reflection"). These APIs are useful for a bunch of things, and in particular they are used to implement text stringification of messages (e.g. for debugging) and JSON conversion. I think some code is also generated in the .c++

Re: [capnproto] How to get server-side implementation from client capability ?

2021-04-28 Thread 'Kenton Varda' via Cap'n Proto
I would say that when you are distributing a precompiled binary copy of your library, you should distribute the `.capnp.h` header along with it, if you want programs using your library to be able to use that type. Those programs will need to link against the exact version of the schema that you

Re: [capnproto] How to get server-side implementation from client capability ?

2021-04-26 Thread 'Kenton Varda' via Cap'n Proto
You need your `SubscriberImpl` to hold a regular C++ pointer back to the `MediaLibraryImpl` that created it, so that you can access its `publishers_` set through that pointer. You can pass the pointer to `SubscriberImpl`'s constructor. You'll also want to make sure the pointer doesn't become a

Re: [capnproto] How to get server-side implementation from client capability ?

2021-04-26 Thread 'Kenton Varda' via Cap'n Proto
Hi Théophile, Take a look at CapabilityServerSet: https://github.com/capnproto/capnproto/blob/6b5bcc2c6e954bc6e167ac581eb628e5a462a469/c++/src/capnp/capability.h#L607-L633 When you convert your `Server` object to a `Client`, you need to do it using a CapabilityServerSet. Later, you can use the

Re: [capnproto] Memory issues with captain proto structs?

2021-03-31 Thread 'Kenton Varda' via Cap'n Proto
Hi Brando, It's hard for us to guess what might be the problem without seeing more code. -Kenton On Tue, Mar 30, 2021 at 12:56 PM Brando Miranda wrote: > Hi, > > I am doing machine learning with captain proto (because captain proto is > good at communicating between python and different

Re: [capnproto] C++: passing around a struct, and Builder<->Reader

2021-03-30 Thread 'Kenton Varda' via Cap'n Proto
FWIW someone (not me) wrote this plugin to generate POCS: https://github.com/nickolasrossi/podgen I haven't used it nor carefully reviewed it yet, but based on the readme it looks pretty thorough. Note it uses std types rather than KJ types which may or may not be to your liking. -Kenton On

Re: [capnproto] C++: passing around a struct, and Builder<->Reader

2021-03-29 Thread 'Kenton Varda' via Cap'n Proto
Hi Topher, Unfortunately, Cap'n Proto is not well suited to representing mutable data structures in-memory. Generally, MessageReaders are read-only, and MessageBuilders are write-only. While technically you can modify builders over time, the memory allocation patterns this tends to lead to are

Re: [capnproto] getTimer in RPC & transparent proxy?

2021-03-25 Thread 'Kenton Varda' via Cap'n Proto
On Thu, Mar 25, 2021 at 11:38 AM pepij...@gmail.com wrote: > > Hey, > > Another one of these things that's probably good design but somewhat > puzzling. > How do I obtain a timer in my RPC server? > I found you need to get them from the async IO provider. > The easyrpc server has a method to get

Re: [capnproto] Re: CPU-bound RPC command & optional API

2021-03-22 Thread 'Kenton Varda' via Cap'n Proto
Yeah, you probably need to recompile the python module against the new version of Cap'n Proto. Unfortunately, it's very hard to remain ABI-compatible across versions in C++, so KJ and Cap'n Proto don't try. -Kenton On Sun, Mar 21, 2021 at 1:13 PM pepijn de vos wrote: > I rewrote my queue for

Re: [capnproto] Re: CPU-bound RPC command & optional API

2021-03-19 Thread 'Kenton Varda' via Cap'n Proto
Yes, Executor is a good way to communicate between threads. FWIW you might also check out newCrossThreadPromiseAndFulfiller(), which was added very recently. Sometimes it's a better fit than kj::Executor. "Level 2" turns out to be something that can't really be built into libcapnp itself because

Re: [capnproto] Wikipedia Draft Article

2021-03-16 Thread 'Kenton Varda' via Cap'n Proto
On Tue, Mar 16, 2021 at 2:26 PM Zach Lym wrote: > I might say: "As of March 2021, three-party handoff and capability equality -- two advanced-but-important features of CapTP -- are not yet supported by Cap'n Proto." >>> >>> That sounds a bit too massaged. I think it should flatly

Re: [capnproto] Wikipedia Draft Article

2021-03-16 Thread 'Kenton Varda' via Cap'n Proto
On Tue, Mar 16, 2021 at 12:13 AM Zach Lym wrote: > Is that last sentence CC0 licensed? > Sure, feel free to reuse anything I wrote verbatim. CC0 license, or public domain, whichever you prefer. > I might say: "As of March 2021, three-party handoff and capability >> equality -- two

Re: [capnproto] Wikipedia Draft Article

2021-03-15 Thread 'Kenton Varda' via Cap'n Proto
Hi Zach, Thanks for doing this! A few nitpicks: > the former maintainer of Google's popular Protocol Buffers RPC framework Technically Protocol Buffers is a serialization framework, not RPC. gRPC and Stubby are RPC frameworks built on Protobuf, but I never worked on either of them. > #@n

Re: [capnproto] How to write to a file from an RPC server?

2021-03-03 Thread 'Kenton Varda' via Cap'n Proto
Hmm when I say "async I/O" I mean the same thing as "non-blocking I/O". I haven't worked in C#, but have done some stuff with the Win32 C API, where they call it "overlapped" I/O. -Kenton On Wed, Mar 3, 2021 at 3:54 PM John Demme wrote: > I think in Windows, async IO is different from

Re: [capnproto] How to write to a file from an RPC server?

2021-03-03 Thread 'Kenton Varda' via Cap'n Proto
On Wed, Mar 3, 2021 at 2:11 PM John Demme wrote: > Some operating systems (*cough* Windows *cough*) don't support > non-blocking file (or inter-process pipe) I/O without some unreliable > hacks. AFAICT. > Hate to say it, but my understanding is that Windows has much more robust async filesystem

Re: [capnproto] How to write to a file from an RPC server?

2021-03-03 Thread 'Kenton Varda' via Cap'n Proto
Hi Pepijn, The full comment you refer to says: // DO NOT CALL THIS except at the top level of your program, e.g. in main(). Anywhere else, you // should instead have your caller pass in a Filesystem object, or a specific Directory object, // or whatever it is that your code needs. This ensures

Re: [capnproto] Re: Premature EOF error with List of List of struct implementation under VS2017/x64

2021-02-21 Thread 'Kenton Varda' via Cap'n Proto
Hi Yojiro, It would be best if you could reduce the problem to a minimal self-contained test case. If your program is multiple files then it is probably too big for us to look at. All I really need is for you to add a main() function to the code you sent before, so I can see exactly the code that

Re: [capnproto] Re: Premature EOF error with List of List of struct implementation under VS2017/x64

2021-02-19 Thread 'Kenton Varda' via Cap'n Proto
Hi, Sorry for the slow reply. Just going to throw out a guess here, but seeing that you're using windows, did you make sure to use O_BINARY when opening the file descriptor? If you didn't, then the C library will silently corrupt your data by converting line endings. If that's not it, then

Re: [capnproto] Mixing event loops

2021-02-11 Thread 'Kenton Varda' via Cap'n Proto
Hi Owen, KJ is actually designed to allow its event loop to sit on top of some other event loop, for this very reason. But, it's not easy. You have to implement a custom subclass of `kj::EventPort` and probably all of the interfaces (that you care about) in `kj/async-io.h`. The node-capnp

Re: [capnproto] Inter process communication used in capnp

2021-02-05 Thread 'Kenton Varda' via Cap'n Proto
Sorry, I really don't know since I haven't tried it yet. There's a lot of potential to eliminate copies and syscalls here, but how much impact that really makes would of course depend on what else your system is doing. (Note that I go by "Kenton", not "Ken"...) -Kenton On Fri, Feb 5, 2021 at

Re: [capnproto] Test failure on AsyncIo/CapabilityPipeBlockedSendStream

2021-02-01 Thread 'Kenton Varda' via Cap'n Proto
Ah, cool. FWIW, this is still just an issue with the design of this particular test case. Having socket buffer set large should be no problem in general. I suppose this test case should probably be rethought but I'll probably wait for a second complaint before I mess with it. -Kenton On Mon,

Re: [capnproto] Test failure on AsyncIo/CapabilityPipeBlockedSendStream

2021-02-01 Thread 'Kenton Varda' via Cap'n Proto
Hmm, well I'm not sure what to say here. Maybe your unix sockets have larger buffers for some reason? Are you running a weird kernel version? What architecture is this? In any case, this error doesn't indicate a problem with Cap'n Proto, only a problem with the test, so you should be safe

Re: [capnproto] Test failure on AsyncIo/CapabilityPipeBlockedSendStream

2021-02-01 Thread 'Kenton Varda' via Cap'n Proto
It looks like this error occurs when the number of file descriptors sent on a unix socket exceeds `ulimit -n`. Admittedly, the test tries to send lots of file descriptors over a unix socket, stopping only when the socket buffer seems to be full. So depending on the system configuration, it could

Re: [capnproto] Test failure on AsyncIo/CapabilityPipeBlockedSendStream

2021-02-01 Thread 'Kenton Varda' via Cap'n Proto
Hi John, Was there anything in the log before the fail line? Usually there should be some sort of error message describing what went wrong, and maybe a stack trace. -Kenton On Mon, Feb 1, 2021 at 3:53 PM John Wang wrote: > I'm seeing a failure on the following test. Did a quick search of the

Re: [capnproto] Capnproto RPC, processing of IPC calls

2020-12-07 Thread 'Kenton Varda' via Cap'n Proto
(Dupe message happened because one got stuck in the moderation queue.) On Mon, Dec 7, 2020 at 10:20 AM Lena Lötter wrote: > Thanks for the quick response and direction. The example comes in handy as > I opted for the longer route. > > On Wednesday, December 2, 2020 at 10:56:13 PM UTC+2

Re: [capnproto] Capnproto RPC for Java - progress

2020-12-02 Thread 'Kenton Varda' via Cap'n Proto
Awesome, I'm very glad to hear you've been paying attention to ordering -- it's tricky to get right but turns out to be very important. I'm hoping David weighs in at some point and we can decide whether it makes sense to merge this back into the main repo. -Kenton On Tue, Dec 1, 2020 at 7:23 AM

Re: [capnproto] Capnproto RPC, processing of IPC calls

2020-12-02 Thread 'Kenton Varda' via Cap'n Proto
Hi Lena, `WaitScope` has a method called `poll()` which checks for new I/O and handles it without blocking. Maybe you can get away with calling that from time to time. If that doesn't work, then it sounds like you may need to implement your own kj::EventPort to use in place of kj::UnixEventPort.

Re: [capnproto] Load balancing and metrics

2020-11-30 Thread 'Kenton Varda' via Cap'n Proto
Yes, it's straightforward to implement a load-balancing proxy server that redirects each call to a different back-end. To do this in C++, you'd implement capnp::Capability::Server, implementing the generic `dispatchCall()` method such that it chooses a back-end and then uses `context.tailCall()`

Re: [capnproto] Capnproto RPC for Java - progress

2020-11-30 Thread 'Kenton Varda' via Cap'n Proto
This is great! Out of curiosity, do you have a specific use case you're building for? Do you have any RPC-specific documentation or examples yet? I notice this is implemented as a fork of the existing capnproto-java. Did you have to make changes to the existing code? Is the API compatible? I

Re: [capnproto] Load balancing and metrics

2020-11-30 Thread 'Kenton Varda' via Cap'n Proto
Hi Kevin, You are correct that if you avoid passing interface types in your RPC messages, you will never have the opportunity to take advantage of promise pipelining, at least as the protocol is defined today. But I'm not sure I understand the motivation for your question. Why do you want a

Re: [capnproto] Schema Refactoring and Unique IDs

2020-11-27 Thread 'Kenton Varda' via Cap'n Proto
On Wed, Nov 25, 2020 at 3:43 PM Matt Stern wrote: > When I try to run capnp compile, I get the following: > > error: Import failed: /capnp/java.capnp > You will need to specify the same -I flags (import path) that you normally specify to `capnp compile` when running the Java code generator. If

Re: [capnproto] Schema Refactoring and Unique IDs

2020-11-25 Thread 'Kenton Varda' via Cap'n Proto
Erin is correct. For #2, the command-line syntax to have the compiler echo back capnp format (with all type IDs defined explicitly) is: capnp compile -ocapnp foo.capnp This of course requires that `capnp` and the generator plugin `capnpc-capnp` are in your $PATH, which they should be after

Re: [capnproto] Build error in dockcross/linux-arm64 Docker

2020-11-16 Thread 'Kenton Varda' via Cap'n Proto
Hi Sundar, Note that the step that is failing is when the build tries to invoke the capnp binary that it just built -- so it would be invoking an aarch64 binary. Does this environment use qemu-user to emulate aarch64? I've noticed in the past that qemu-user has a lot of bugs emulating certain

Re: [capnproto] Build error in dockcross/linux-arm64 Docker

2020-11-15 Thread 'Kenton Varda' via Cap'n Proto
Hi Sundar, Can you tell us the set of commands that led to this? I.e. what was your command-line for `configure` and `make`? And what directory did you run them in, relative to the source directory? -Kenton On Sun, Nov 15, 2020 at 6:22 PM Sundar Palani wrote: > I have the following error

Re: [capnproto] Segfault in on Ubuntu 14.04 32-bit

2020-11-09 Thread 'Kenton Varda' via Cap'n Proto
Yep, that looks like a smoking gun to me. -Kenton On Mon, Nov 9, 2020 at 2:41 PM Zachary Dremann wrote: > This looks related: > https://lists.debian.org/debian-glibc/2014/01/msg00023.html > > I rebuilt everything in the same way on an ubuntu 16.04 VM, and I don't > get the segfault. I don't

Re: [capnproto] Segfault in on Ubuntu 14.04 32-bit

2020-11-06 Thread 'Kenton Varda' via Cap'n Proto
Hi Zachary, It looks like the segfault is happening in glibc's backtrace() function while gathering the stack trace for an exception. An exception is actually expected here (the test case tests throwing an exception). Since it happened specifically in fiber-related tests, my guess is that glibc's

Re: [capnproto] enum brands?

2020-10-30 Thread 'Kenton Varda' via Cap'n Proto
I probably did it for consistency. It's true that, at least at present, it's not in any way meaningful for an enum to be parameterized. Moreover, you cannot declare parameters in an enum declaration. However, an enum could be nested within a parent scope that is parameterized, and in theory

Re: [capnproto] Middleware & Attaching metadata to calls/returns?

2020-10-19 Thread 'Kenton Varda' via Cap'n Proto
Hi Erin, In Cloudflare Workers, we've had some success attaching certain kinds of metadata by having a wrapper object which is called first, like: interface Endpoint { startCall @0 (metadata :Metadata) -> (dispatcher :CallDispatcher); } interface CallDispatcher { foo @0

Re: [capnproto] How to implement streaming from server to client?

2020-10-19 Thread 'Kenton Varda' via Cap'n Proto
Hi Dmitriy, I had a long set of answers typed up for you, but then I came to your last paragraph: > PLEASE DO NOT BE LIKE STUPID AMERICANS. Do not answer for any question > with "Yes". US citizens likes answer for complex question with non related > answer. > Please be as more verbose as you

Re: [capnproto] Assert in exception.c++ triggering only in ASAN

2020-10-18 Thread 'Kenton Varda' via Cap'n Proto
I guess this means that under ASAN, the stack is not allocated in the traditional way. For some reason there's a 1MB offset between where the ExceptionCallback was allocated and where a local variable inside its constructor was allocated. The check there is pretty hacky and probably technically

Re: [capnproto] TSAN faiure

2020-10-13 Thread 'Kenton Varda' via Cap'n Proto
On Tue, Oct 13, 2020 at 5:15 AM Erin Shepherd wrote: > > I think that ThreadSanitizer is having trouble recognizing that the > initialization of `brokenCapFactory` is thread-safe, due to the awkward way > in which it is initialized. It may end up being initialized by multiple > threads, but all

Re: [capnproto] TSAN faiure

2020-10-10 Thread 'Kenton Varda' via Cap'n Proto
hat's carefully tuned > > On Fri, Oct 9, 2020 at 10:00 AM 'Kenton Varda' via Cap'n Proto < > capnproto@googlegroups.com> wrote: > >> It looks like the writes are done atomically, but the reads aren't. TSAN >> is right to be suspicious of this. In practice there is

Re: [capnproto] TSAN faiure

2020-10-09 Thread 'Kenton Varda' via Cap'n Proto
It looks like the writes are done atomically, but the reads aren't. TSAN is right to be suspicious of this. In practice there is no bug here, but showing that requires higher-level reasoning that TSAN wouldn't be expected to figure out. -Kenton On Fri, Oct 9, 2020 at 11:52 AM Vitali Lovich

Re: [capnproto] TSAN faiure

2020-10-09 Thread 'Kenton Varda' via Cap'n Proto
I think that ThreadSanitizer is having trouble recognizing that the initialization of `brokenCapFactory` is thread-safe, due to the awkward way in which it is initialized. It may end up being initialized by multiple threads, but all threads will initialize it to the same value, hence no atomics

Re: [capnproto] Implementing Level-3 Protocol (in Go)

2020-10-01 Thread 'Kenton Varda' via Cap'n Proto
Was someone going to send out a meeting invite or something? On Wed, Sep 30, 2020, 4:11 PM Louis Thibault wrote: > Alright, it seems like everyone is eager to meet this Friday, so let’s do > that. :) > > See you all on Friday, as initially planned. > > - Louis > > On Sep 30, 2020, at 17:08,

Re: [capnproto] Standardized schema interchange

2020-09-30 Thread 'Kenton Varda' via Cap'n Proto
On Wed, Sep 23, 2020 at 11:52 PM Ryan Patterson wrote: > Hi Kenton, thanks for weighing in! > > > I don't think the generated code would implement the getSchema() RPC > directly for each type. Instead I think I'd add a new method to the > `Capability::Server` and `ClientHook` like: > > I think

Re: [capnproto] How to keep some incoming messages as is and reuse?

2020-09-30 Thread 'Kenton Varda' via Cap'n Proto
Hi Ambrase, copyToUnchecked() does not write a segment table at the beginning of the message. That's probably why you're seeing a message 8 bytes shorter. copyToUnchecked() and readMessageUnchecked() are difficult to use correctly (and are insecure when used incorrectly). I strongly recommend

Re: [capnproto] Implementing Level-3 Protocol (in Go)

2020-09-30 Thread 'Kenton Varda' via Cap'n Proto
I'm still happy to chat this Friday. -Kenton On Wed, Sep 30, 2020 at 3:41 PM Louis Thibault wrote: > 100% agree — let’s not lose momentum. I’ll send out another invite when > we hear back from Ross. > > Cheers, > Louis > > On Sep 30, 2020, at 16:27, Ian Denhardt wrote: > > I'm okay

Re: [capnproto] Implementing Level-3 Protocol (in Go)

2020-09-30 Thread 'Kenton Varda' via Cap'n Proto
Sorry for the very slow response this time. We had a big launch on Monday and I haven't had much time to reply to e-mail. Incidentally that launch is vaguely relevant here. https://blog.cloudflare.com/introducing-workers-durable-objects/ We built the infrastructure for this on Cap'n Proto. When

Re: [capnproto] Implementing Level-3 Protocol (in Go)

2020-09-23 Thread 'Kenton Varda' via Cap'n Proto
Hi Louis, This is somewhat embarrassing, but there is no level 3 implementation of Cap'n Proto today, in any language. It's likely that when I get around to implementing it in C++, the protocol spec will change to solve unanticipated problems. As such, trying to implement the spec as-written in

Re: [capnproto] Can't compile 0.8 release with -Wshadow -Werror

2020-09-14 Thread 'Kenton Varda' via Cap'n Proto
Hi Jeremy, The `Which` enums are intentionally not `enum class` because I wanted the enumerants to appear as constants in the containing class, as this enum actually represents the union tags for the containing class, so the `Which` part is sort of superfluous. All other enums, though, are

Re: [capnproto] Re: Use with TCP socket, C++ -> pycapnp, no RPC

2020-09-10 Thread 'Kenton Varda' via Cap'n Proto
Hi Robert, > message has too many segments error. This basically means the data is corrupted somehow. You'll need to track down why the bytes you're receiving are not the same as the bytes you are sending. To debug, you might try reading the bytes into a regular byte buffer and logging them, and

Re: [capnproto] Translate RESTFul to capnp

2020-08-24 Thread 'Kenton Varda' via Cap'n Proto
Hi Mostafa, Nope, we don't have that yet. We have JsonCodec to translate JSON to Cap'n Proto, but no one has put it together into an HTTP service just yet. It's more complicated for Cap'n Proto compared to gRPC because of capabilities and promise pipelining. There isn't an obvious way to expose

Re: [capnproto] request for simple interface inheritance?

2020-08-14 Thread 'Kenton Varda' via Cap'n Proto
Hmm, would `Bar` be allowed to be used as a type in schemas? If so, when you receive an arbitrary `Bar` on the wire, how would you figure out which layout to use? If `Bar` is not allowed on the wire then I can imagine how to implement this, but I think a lot of people would be surprised by such a

Re: [capnproto] How to compress/use packing with RPC

2020-08-03 Thread 'Kenton Varda' via Cap'n Proto
It's true, currently TwoPartyVatNetwork doesn't feature any flags to enable these. You could write a custom VatNetwork implementation that does it. I'd be happy to accept a PR adding the features to TwoPartyVatNetwork. -Kenton On Thu, Jul 30, 2020 at 7:57 PM wrote: > I can't seem to find out

Re: [capnproto] Easiest way to connect two kj::AsyncIoStream?

2020-07-30 Thread 'Kenton Varda' via Cap'n Proto
Hey, sorry for the delayed response, it's been a really busy time. And TBH I'm not sure I really have an answer. The Windows async bindings are definitely very very different from the Unix ones, since Windows uses overlapped I/O and Unix uses readiness I/O and, well, they work totally

Re: [capnproto] Writing to a boost gzip filter stream

2020-07-20 Thread 'Kenton Varda' via Cap'n Proto
Hi Emyr, You can use capnp::writeMessage() to write to any kind of stream. https://github.com/capnproto/capnproto/blob/7e2f70f354e2ca3d9de73ddaf974408ddbd98866/c++/src/capnp/serialize.h#L162 This function writes to a `kj::OutputStream`, which is an abstract interface that you can implement any

Re: [capnproto] request for simple interface inheritance?

2020-07-20 Thread 'Kenton Varda' via Cap'n Proto
If you have a struct Foo whose fields are a strict superset of some other struct Bar (with matching field numbers and everything), then you can convert one to the other like: Foo::Reader foo = ...; Bar::Reader bar = capnp::AnyStruct::Reader(foo).as(); (Or vice versa.) But the big

Re: [capnproto] Integrating KJ with other systems

2020-07-13 Thread 'Kenton Varda' via Cap'n Proto
On Mon, Jul 6, 2020 at 10:54 AM Vitali Lovich wrote: > I think a potential middle ground for this might be to have the stream > give you the fd and a fulfiller. Any I/O operations on the stream then are > blocked from executing until after the promise to return the FD is > fulfilled. > That's

Re: [capnproto] Easiest way to connect two kj::AsyncIoStream?

2020-07-13 Thread 'Kenton Varda' via Cap'n Proto
Hmm I don't understand what you're doing with paf.promise.fork(), it looks like you're merely attaching it to another promise, not actually waiting on it or anything. I am a little concerned about your use of shutdownWrite() / abortRead() in a way that assumes these functions will cause any

Re: [capnproto] Integrating KJ with other systems

2020-07-06 Thread 'Kenton Varda' via Cap'n Proto
On Mon, Jul 6, 2020 at 10:22 AM Vitali Lovich wrote: > I don't feel great about the duplication approach (or allowing retrieving > the raw handle directly) as it can be subtly tricky to actually use > correctly. For example, if you dup & make the dup'ed FD blocking, that will > impact correct

Re: [capnproto] Integrating KJ with other systems

2020-07-06 Thread 'Kenton Varda' via Cap'n Proto
On Mon, Jul 6, 2020 at 9:28 AM Vitali Lovich wrote: > Yeah, that's what I figured. The KJ API is s much more user friendly >>> though :). It would be cool to be able to consume the raw file >>> descriptor/socket out of the pipes the LowlevelIoProvider constructs to >>> simplify code/error

Re: [capnproto] Easiest way to connect two kj::AsyncIoStream?

2020-07-06 Thread 'Kenton Varda' via Cap'n Proto
Almost. Two issues I can think of: 1. After each pump completes, you probably want to call shutdownWrite() to propagate the EOF. 2. `joinPromises()` always waits for all promises to resolve. I've often found that this turns out to be the wrong behavior when one of the joined promises throws an

Re: [capnproto] Integrating KJ with other systems

2020-07-06 Thread 'Kenton Varda' via Cap'n Proto
On Mon, Jul 6, 2020 at 8:02 AM Vitali Lovich wrote: > On Sun, Jul 5, 2020 at 3:02 PM Kenton Varda wrote: > >> On Sun, Jul 5, 2020 at 1:48 PM Vitali Lovich wrote: >> >>> Yeah, so I'm still working through getting executeSync working. I do >>> wish executeAsync would still be able to do the

Re: [capnproto] Integrating KJ with other systems

2020-07-05 Thread 'Kenton Varda' via Cap'n Proto
On Sun, Jul 5, 2020 at 1:48 PM Vitali Lovich wrote: > Yeah, so I'm still working through getting executeSync working. I do wish > executeAsync would still be able to do the right thing when invoked from > off-thread to avoid creating artificial synchronization points, but I can > see it being

Re: [capnproto] Integrating KJ with other systems

2020-07-05 Thread 'Kenton Varda' via Cap'n Proto
I posted an answer on Stack Overflow. On Fri, Jul 3, 2020 at 10:47 PM Vitali Lovich wrote: > I'm trying to integrate cap'n'proto with surrounding threads that don't > use KJ event loops. On Android it's easier because I have an event loop to > adapt to with ALooper (albeit it's crashing -

Re: [capnproto] Capnproto RPC messaging with lockfree ring buffer between threads

2020-07-01 Thread 'Kenton Varda' via Cap'n Proto
In fact, it is technically possible to use interfaces independently of the RPC system. Study capnp/capability.h to see the interfaces you'd need to define. However, I guess you'd end up repeating a lot of the work of the RPC implementation. So I don't recommend that. I guess it probably makes the

Re: [capnproto] Capnproto RPC messaging with lockfree ring buffer between threads

2020-07-01 Thread 'Kenton Varda' via Cap'n Proto
Hi Dmitry, You should be able to accomplish this by writing a custom implementation of VatNetwork. You can look at TwoPartyVatNetwork for an example. That said, if you're thinking about optimizations like lock-free ring buffers, you may find that the RPC system itself adds too much overhead for

Re: [capnproto] Implementing generic servers using pycapnp

2020-06-22 Thread 'Kenton Varda' via Cap'n Proto
Hi Vaci, I suspect pycapnp currently lacks support for specifying generic type parameters. You might want to file an issue on pycapnp's github about that. However, I suspect you can work around it by defining the type specializations you need in a .capnp file (which could be a private file used

Re: [capnproto] How do I detect when a remote capability that is passed to 3rd-party is released by 3rd-party?

2020-06-19 Thread 'Kenton Varda' via Cap'n Proto
Hi Christian, It's intentional that you can't see the event in this case. Eventually, we expect that the third server will be able to form a direct connection to the first server here, so the second server is cut out of the loop. There are three design patterns you can use to get what you want

Re: [capnproto] CapnProto C++ as library for parsing textual schemas

2020-06-18 Thread 'Kenton Varda' via Cap'n Proto
`schema.getProto().getNestedNodes()` returns a list of `capnp::schema::Node::NestedNode`, which has `name` and `id` fields. So that gives you the names of the nested declarations, which you can get plug into `getNested()`. It's a little ugly, but in general, the C++ schema classes only have

Re: [capnproto] CapnProto C++ as library for parsing textual schemas

2020-06-18 Thread 'Kenton Varda' via Cap'n Proto
Hi John, `capnp::SchemaParser` will parse capnp files and give you a `capnp::ParsedSchema`, which is like `capnp::Schema` but also has the `getNested()` method which allows you to traverse the whole tree of child nodes. So it's not just the root node. That said, it's true that it's not

<    1   2   3   4   5   >