Re: [capnproto] Accept connection in new process

['Kenton Varda' via Cap'n Proto]

Yeah, C++ error messages are the worst. :(

You kind of have to get used to knowing that "can't convert" errors are
usually because you forgot to kj::mv...

-Kenton

On Tue, Nov 30, 2021 at 10:08 AM pepijn de vos 
wrote:

> Ah I solved it... by not storing sim in a variable. Or probably kj::mv
> would have done the job too.
> It's kinda unfortunate the error you get just tells you it's the wrong
> type, rather than.. hey you should move the thing.
> Onwards!!
> Pepijn
>
> On Tue, Nov 30, 2021 at 4:50 PM pepijn de vos 
> wrote:
>
>> Hi Kenton,
>>
>> I got back to this problem and tried implementing the fork idea but ran
>> into another silly problem where the C++ templates are smarter than me it
>> seems.
>> You recommended to not use the EzRPC server, but there is a great lack of
>> examples of other ways to go about it.
>>
>> I found a Rust example that uses the low-level API but of course not
>> exactly the C++ one:
>> https://github.com/capnproto/capnproto-rust/blob/master/capnp-rpc/examples/calculator/server.rs
>> I found some code that's copy-pasted all over using github search that
>> seems to use the lower level API:
>> https://github.com/Nickan/nickan.github.io/blob/b0eb31c33a3b6720606974f9576e663f3b7852ca/drawio/etc/sandstorm/server.c%2B%2B#L420-L422
>>
>> However, I cannot quite get it to work.
>>
>> kj::AsyncIoContext ctx = kj::setupAsyncIo();
>> auto stream = ctx.lowLevelProvider->wrapSocketFd(clientFd);
>> auto network = capnp::TwoPartyVatNetwork(*stream, capnp::rpc::twoparty::
>> Side::CLIENT);
>> kj::Own fs = kj::newDiskFilesystem();
>> const kj::Directory &dir = fs->getCurrent();
>>
>> auto sim = kj::heap(dir);
>> auto rpc = capnp::makeRpcServer(network, sim);
>>
>> So I set up asyncio, wrap my unix socket FD into a stream, use that to
>> create a network (client or server??)
>> The problem is the call to makeRpcServer. SimulatorImpl is a ::Server
>> subclass.
>> This seems to match the random copy-pasted example, but gives compile
>> errors that it can't convert sim to whatever is expected.
>> I tried reading the source code, which tells me it expects a restorer or
>> a bootstrap, but the restorer is deprecated, but also what EzRPC seems to
>> be using.
>> I tried looking at the tests but that appears to do the same thing, just
>> pass it an owned instance of the implementation??
>>
>> https://github.com/capnproto/capnproto/blob/e5af75ff0513e6d971fa7b48e8108427766e51f9/c%2B%2B/src/capnp/rpc-test.c%2B%2B#L1273-L1279
>> I am at a loss.
>>
>> Pepijn
>>
>>
>> On Thu, May 6, 2021 at 8:22 PM Kenton Varda 
>> wrote:
>>
>>> I don't recommend using EzRPC. That interface turns out to be too
>>> restrictive.
>>>
>>> To set up a KJ event loop, use kj::setupAsyncIo().
>>>
>>> Then to initiate or accept Cap'n Proto RPC connections, use
>>> capnp::TwoPartyClient / capnp::TwoPartyServer.
>>>
>>> For FD passing, create a capnp server object that overrides the virtual
>>> method `kj::Maybe Capability::Server::getFd()` to return an FD number.
>>> Then on the Client objects pointing to that server, you can call `getFd()`
>>> to get that file descriptor -- even from another process, as long as the
>>> connection between the two is a unix domain socket.
>>>
>>> -Kenton
>>>
>>> On Thu, May 6, 2021 at 1:07 PM pepijn de vos 
>>> wrote:
>>>
 Thanks for the suggestions.
 This is still for the simulation server, so each connection is pretty
 expensive anyway, and it's unlikely there will be more in flight than the
 machine has cores available. It might even make sense to have a pool as you
 suggest, and just reject connections if there are no cores available.
 Concern is that you need to supervise these pools in case they crash. With
 fork you can just launch a new process. Any pointers to useful functions
 for this kinds of hacks would be appreciated. I don't have a clue how I'd
 go about obtaining and sending file descriptors over RPC and making new
 servers out of them. I should probably start by studying EzRPC and
 kj::LowLevelAsyncIoProvider.

 Pepijn

 On Thu, May 6, 2021 at 7:48 PM Kenton Varda 
 wrote:

> Ohhh, sorry, I misread. If you haven't started the KJ event loop until
> after the fork, you are probably OK.
>
> You can definitely construct a KJ socket from a raw socket using
> kj::LowLevelAsyncIoProvider.
>
> But if you're OK with the entire connection being handled in the
> isolated process, there are some other options too:
> - Have a pool of processes that are all waiting to accept from the
> same file descriptor. Each connection will only be accepted by one of the
> processes. Make sure that process doesn't accept a new connection until 
> the
> old connection is done.
> - Cap'n Proto supports sending file descriptors between processes in
> RPC messages. So you could have one process that is accepting connections,
> but each time it does, it sends the file descriptor to

Re: [capnproto] Accept connection in new process

[pepijn de vos]

Ah I solved it... by not storing sim in a variable. Or probably kj::mv
would have done the job too.
It's kinda unfortunate the error you get just tells you it's the wrong
type, rather than.. hey you should move the thing.
Onwards!!
Pepijn

On Tue, Nov 30, 2021 at 4:50 PM pepijn de vos  wrote:

> Hi Kenton,
>
> I got back to this problem and tried implementing the fork idea but ran
> into another silly problem where the C++ templates are smarter than me it
> seems.
> You recommended to not use the EzRPC server, but there is a great lack of
> examples of other ways to go about it.
>
> I found a Rust example that uses the low-level API but of course not
> exactly the C++ one:
> https://github.com/capnproto/capnproto-rust/blob/master/capnp-rpc/examples/calculator/server.rs
> I found some code that's copy-pasted all over using github search that
> seems to use the lower level API:
> https://github.com/Nickan/nickan.github.io/blob/b0eb31c33a3b6720606974f9576e663f3b7852ca/drawio/etc/sandstorm/server.c%2B%2B#L420-L422
>
> However, I cannot quite get it to work.
>
> kj::AsyncIoContext ctx = kj::setupAsyncIo();
> auto stream = ctx.lowLevelProvider->wrapSocketFd(clientFd);
> auto network = capnp::TwoPartyVatNetwork(*stream, capnp::rpc::twoparty::
> Side::CLIENT);
> kj::Own fs = kj::newDiskFilesystem();
> const kj::Directory &dir = fs->getCurrent();
>
> auto sim = kj::heap(dir);
> auto rpc = capnp::makeRpcServer(network, sim);
>
> So I set up asyncio, wrap my unix socket FD into a stream, use that to
> create a network (client or server??)
> The problem is the call to makeRpcServer. SimulatorImpl is a ::Server
> subclass.
> This seems to match the random copy-pasted example, but gives compile
> errors that it can't convert sim to whatever is expected.
> I tried reading the source code, which tells me it expects a restorer or a
> bootstrap, but the restorer is deprecated, but also what EzRPC seems to be
> using.
> I tried looking at the tests but that appears to do the same thing, just
> pass it an owned instance of the implementation??
>
> https://github.com/capnproto/capnproto/blob/e5af75ff0513e6d971fa7b48e8108427766e51f9/c%2B%2B/src/capnp/rpc-test.c%2B%2B#L1273-L1279
> I am at a loss.
>
> Pepijn
>
>
> On Thu, May 6, 2021 at 8:22 PM Kenton Varda  wrote:
>
>> I don't recommend using EzRPC. That interface turns out to be too
>> restrictive.
>>
>> To set up a KJ event loop, use kj::setupAsyncIo().
>>
>> Then to initiate or accept Cap'n Proto RPC connections, use
>> capnp::TwoPartyClient / capnp::TwoPartyServer.
>>
>> For FD passing, create a capnp server object that overrides the virtual
>> method `kj::Maybe Capability::Server::getFd()` to return an FD number.
>> Then on the Client objects pointing to that server, you can call `getFd()`
>> to get that file descriptor -- even from another process, as long as the
>> connection between the two is a unix domain socket.
>>
>> -Kenton
>>
>> On Thu, May 6, 2021 at 1:07 PM pepijn de vos 
>> wrote:
>>
>>> Thanks for the suggestions.
>>> This is still for the simulation server, so each connection is pretty
>>> expensive anyway, and it's unlikely there will be more in flight than the
>>> machine has cores available. It might even make sense to have a pool as you
>>> suggest, and just reject connections if there are no cores available.
>>> Concern is that you need to supervise these pools in case they crash. With
>>> fork you can just launch a new process. Any pointers to useful functions
>>> for this kinds of hacks would be appreciated. I don't have a clue how I'd
>>> go about obtaining and sending file descriptors over RPC and making new
>>> servers out of them. I should probably start by studying EzRPC and
>>> kj::LowLevelAsyncIoProvider.
>>>
>>> Pepijn
>>>
>>> On Thu, May 6, 2021 at 7:48 PM Kenton Varda 
>>> wrote:
>>>
 Ohhh, sorry, I misread. If you haven't started the KJ event loop until
 after the fork, you are probably OK.

 You can definitely construct a KJ socket from a raw socket using
 kj::LowLevelAsyncIoProvider.

 But if you're OK with the entire connection being handled in the
 isolated process, there are some other options too:
 - Have a pool of processes that are all waiting to accept from the same
 file descriptor. Each connection will only be accepted by one of the
 processes. Make sure that process doesn't accept a new connection until the
 old connection is done.
 - Cap'n Proto supports sending file descriptors between processes in
 RPC messages. So you could have one process that is accepting connections,
 but each time it does, it sends the file descriptor to some other processes
 via an RPC. Then that process actually handles the connection.

 These options are both more complicated than the simple fork() approach
 you mentioned, so aren't necessarily better. But fork() is pretty expensive
 so it might turn out reusing processes has some performance benefit.

 -Kenton
>>>

Re: [capnproto] Accept connection in new process

[pepijn de vos]

Hi Kenton,

I got back to this problem and tried implementing the fork idea but ran
into another silly problem where the C++ templates are smarter than me it
seems.
You recommended to not use the EzRPC server, but there is a great lack of
examples of other ways to go about it.

I found a Rust example that uses the low-level API but of course not
exactly the C++ one:
https://github.com/capnproto/capnproto-rust/blob/master/capnp-rpc/examples/calculator/server.rs
I found some code that's copy-pasted all over using github search that
seems to use the lower level API:
https://github.com/Nickan/nickan.github.io/blob/b0eb31c33a3b6720606974f9576e663f3b7852ca/drawio/etc/sandstorm/server.c%2B%2B#L420-L422

However, I cannot quite get it to work.

kj::AsyncIoContext ctx = kj::setupAsyncIo();
auto stream = ctx.lowLevelProvider->wrapSocketFd(clientFd);
auto network = capnp::TwoPartyVatNetwork(*stream, capnp::rpc::twoparty::Side
::CLIENT);
kj::Own fs = kj::newDiskFilesystem();
const kj::Directory &dir = fs->getCurrent();

auto sim = kj::heap(dir);
auto rpc = capnp::makeRpcServer(network, sim);

So I set up asyncio, wrap my unix socket FD into a stream, use that to
create a network (client or server??)
The problem is the call to makeRpcServer. SimulatorImpl is a ::Server
subclass.
This seems to match the random copy-pasted example, but gives compile
errors that it can't convert sim to whatever is expected.
I tried reading the source code, which tells me it expects a restorer or a
bootstrap, but the restorer is deprecated, but also what EzRPC seems to be
using.
I tried looking at the tests but that appears to do the same thing, just
pass it an owned instance of the implementation??
https://github.com/capnproto/capnproto/blob/e5af75ff0513e6d971fa7b48e8108427766e51f9/c%2B%2B/src/capnp/rpc-test.c%2B%2B#L1273-L1279
I am at a loss.

Pepijn


On Thu, May 6, 2021 at 8:22 PM Kenton Varda  wrote:

> I don't recommend using EzRPC. That interface turns out to be too
> restrictive.
>
> To set up a KJ event loop, use kj::setupAsyncIo().
>
> Then to initiate or accept Cap'n Proto RPC connections, use
> capnp::TwoPartyClient / capnp::TwoPartyServer.
>
> For FD passing, create a capnp server object that overrides the virtual
> method `kj::Maybe Capability::Server::getFd()` to return an FD number.
> Then on the Client objects pointing to that server, you can call `getFd()`
> to get that file descriptor -- even from another process, as long as the
> connection between the two is a unix domain socket.
>
> -Kenton
>
> On Thu, May 6, 2021 at 1:07 PM pepijn de vos 
> wrote:
>
>> Thanks for the suggestions.
>> This is still for the simulation server, so each connection is pretty
>> expensive anyway, and it's unlikely there will be more in flight than the
>> machine has cores available. It might even make sense to have a pool as you
>> suggest, and just reject connections if there are no cores available.
>> Concern is that you need to supervise these pools in case they crash. With
>> fork you can just launch a new process. Any pointers to useful functions
>> for this kinds of hacks would be appreciated. I don't have a clue how I'd
>> go about obtaining and sending file descriptors over RPC and making new
>> servers out of them. I should probably start by studying EzRPC and
>> kj::LowLevelAsyncIoProvider.
>>
>> Pepijn
>>
>> On Thu, May 6, 2021 at 7:48 PM Kenton Varda 
>> wrote:
>>
>>> Ohhh, sorry, I misread. If you haven't started the KJ event loop until
>>> after the fork, you are probably OK.
>>>
>>> You can definitely construct a KJ socket from a raw socket using
>>> kj::LowLevelAsyncIoProvider.
>>>
>>> But if you're OK with the entire connection being handled in the
>>> isolated process, there are some other options too:
>>> - Have a pool of processes that are all waiting to accept from the same
>>> file descriptor. Each connection will only be accepted by one of the
>>> processes. Make sure that process doesn't accept a new connection until the
>>> old connection is done.
>>> - Cap'n Proto supports sending file descriptors between processes in RPC
>>> messages. So you could have one process that is accepting connections, but
>>> each time it does, it sends the file descriptor to some other processes via
>>> an RPC. Then that process actually handles the connection.
>>>
>>> These options are both more complicated than the simple fork() approach
>>> you mentioned, so aren't necessarily better. But fork() is pretty expensive
>>> so it might turn out reusing processes has some performance benefit.
>>>
>>> -Kenton
>>>
>>> On Thu, May 6, 2021 at 12:39 PM pepijn de vos 
>>> wrote:
>>>
 Hi Kenton,

 Agree forking an eventloop will lead to disaster, which is why I
 suggested having a blocking accept loop, and then fork and start the event
 loop in the new process. But I'm not sure if it's at all possible to
 convert a raw unix socket to a jk socket and then give it to an eventloop
 to handle the connection.

 RPC t

Re: [capnproto] Accept connection in new process

['Kenton Varda' via Cap'n Proto]

Hi Pepijn,

Yeah, forking won't work well. I don't recommend that. One problem is that
if you fork() with an event loop running, both the parent and child
processes will end up with the same epoll FD and so will receive each
other's events, which will likely lead to chaos.

So I recommend spawning a subprocess and talking to it using RPC over a
unix socket. Yes, it adds some overhead, but really not that much.

-Kenton

On Thu, May 6, 2021 at 2:39 AM pepij...@gmail.com 
wrote:

> Hey,
>
> I'm writing an API for some shared libraries that are written with a
> "binary" use in mind.
> They use a ton of global state and are not thread safe.
> Of course as a server, multiple users could try to use the server at the
> same time, these libraries can't deal with that. What's more they also like
> to segfault sometimes.
>
> So what I'd like to do is have these shared libraries completely in their
> own address space.
> What would be the best way to do that?
>
> One thing I thought could work is have a blocking accept loop, and then
> fork the handling of the connection. Not sure how this would work with kj.
>
> The other solution would be that you launch a process in the method
> handler that runs its own capnp server, and then proxy calls to it. Seems
> like it adds more overhead.
>
> Regards,
> Pepijn
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/99fcf2cf-e6e4-4918-8734-22b7f85cfb85n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQmfWx6kk1Ejdr3bsfJQSxAw-aF%2BxA%2BBuAj4QRaj2pZ_uA%40mail.gmail.com.

[capnproto] Accept connection in new process

[pepij...@gmail.com]

Hey,

I'm writing an API for some shared libraries that are written with a 
"binary" use in mind.
They use a ton of global state and are not thread safe.
Of course as a server, multiple users could try to use the server at the 
same time, these libraries can't deal with that. What's more they also like 
to segfault sometimes.

So what I'd like to do is have these shared libraries completely in their 
own address space.
What would be the best way to do that?

One thing I thought could work is have a blocking accept loop, and then 
fork the handling of the connection. Not sure how this would work with kj.

The other solution would be that you launch a process in the method handler 
that runs its own capnp server, and then proxy calls to it. Seems like it 
adds more overhead.

Regards,
Pepijn

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/99fcf2cf-e6e4-4918-8734-22b7f85cfb85n%40googlegroups.com.