Re: [capnproto] Server process stops accepting connections

['Kenton Varda' via Cap'n Proto]

Hi Jens,

Sorry for the slow reply, I've been on leave.

In general, KJ async I/O objects are tied to the thread / event loop where
they were created. This means you cannot pass an AsyncIoStream between
threads. You can, however, pass a file descriptor. If you are careful, you
could tear down the AsyncIoStream wrapping your socket, pass the socket
file descriptor to another thread, and construct a new AsyncIoStream around
it there.

-Kenton

On Wed, Nov 9, 2022 at 12:19 PM Jens Alfke  wrote:

>
> On Nov 9, 2022, at 6:47 AM, 'Kenton Varda' via Cap'n Proto <
> capnproto@googlegroups.com> wrote:
>
> Have you tested whether your server is able to accept concurrent
> connections normally? E.g. if you open a connection with telnet or
> something without sending any bytes, leave that open, and then try to use
> your server, does it work?
>
>
> *Thanks! That was indeed the problem.* The telnet test showed my server
> would accept multiple connections at once, but would only authenticate one
> of them at a time, so a client that didn’t send the right handshake would
> block others.
>
> (I’m using custom code, largely copied from rpc.c++ and tls.c++, but
> substituting the SecretHandshake protocol (from Scuttlebutt) for TLS. When
> I adapted TlsConnectionReceiver::acceptLoop, I got things in the wrong
> order, so the tail-recursion back to acceptLoop didn’t occur until the
> secure handshake finished.)
>
> I’ve fixed it now. Next question: *if I want to run client connections on
> multiple threads*, I presume that after the accept-and-handshake
> finishes, I would create a new thread with an event loop, then use an
> Executor to pass the AsyncIoStream to it, and then do the magic RPC setup
> on that thread. On disconnect I’d either exit the thread or return it to a
> pool for reuse. Most of this seems like boilerplate … are there any working
> examples I could steal from?
>
> (Why would I want multiple threads when CapnP is async? Because my actual
> RPC methods do SQLite queries, which are blocking and sometimes slow, and
> I’d rather not create async wrappers for my entire complicated database
> API.)
>
> —Jens
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQ%3DZfFUQYcFzKfUtJ57KuXH5Jpbd2HqX4Hbh8wLGdW8-Hw%40mail.gmail.com.

Re: [capnproto] capnp::messageToFlatArray to pre-allocated buffer?

['Kenton Varda' via Cap'n Proto]

Hi Hui,

Sorry for the slow reply, I was on leave.

messageToFlatArray() is a convenience function for when you don't mind
allocation and copying. If you want more control over serialization, you
can use MessageBuilder::getSegmentsForOutput() to get direct pointers to
the message's underlying buffer space, without copying anything. You can
then serialize these segments as you see fit. Note that you are responsible
for tracking the segment boundaries in this case. On the receiving end,
pass the same array-of-arrays into SegmentArrayMessageReader to parse the
message. Again, this doesn't perform any allocation or copies; that's up to
you to manage.

Another option is to implement your own kj::OutputStream, and then use
capnp::writeMessage() to write a message to the stream. This will always
write the entire message using one single call to write() on the underlying
stream, passing an array-of-arrays that point into the original message
buffer -- again, no copies. You could then copy into your scratch buffer or
do whatever else you want. An advantage of this approach (using
OutputStream instead of getSegmentsForOutput()) is that writeMessage() will
also build a segment table for you and include that in the output, so on
the receiving end you can use FlatArrayMessageReader instead of
SegmentArrayMessageReader.

-Kenton

On Sat, Nov 5, 2022 at 9:29 PM Hui min  wrote:

> Hi CapnProto Team,
>
> I understand the API to obtain a flat buffer of the built message is
> capnp::messageToFlatArray.
>
> However, the current API forces you to save the buffer on a returned
> kj::Array words object. Is there a way I could pass in a
> pre-allocated buffer, and let CapnProto to use that for serialisation
> (which essentially is creating a header + concat all the segments?)?
>
> Currently, some of the API has to be implemented in this way, when
> pre-allocated buffer is present. This introduced an extra copy and
> serialisation time.
>
> https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/include/ecal/msg/capnproto/publisher.h#L147-L150
>
> A good kind of API would be like this:
>
> https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/include/ecal/msg/protobuf/publisher.h#L162
>
> Thanks in advance!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/70bbe8b2-c06d-495f-ac6d-6eedecbdd305n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkZ3uQPfobVzqZnOWsn6m-AhW3kt-tGZxAoFTvg_nK7vw%40mail.gmail.com.