Hi Gym,
There's no way to do true zero-copy from one RPC to another. You will have
to copy. And yes, if the outgoing messages have to be built after the
incoming RPC has completed, then you have to make two copies, once into a
temporary in-memory structure.
Note that, if you have a complex structure, using `capnp::clone(reader)` to
create the temporary copy is likely more efficient than using tree of plain
C++ objects, as `capnp::clone` makes sure to do only one allocation where
the entire message is stored together. Also, copying between capnp messages
is generally very efficient compared to converting to a different style of
representation and back.
If your message contains very large Text or Data values, you might be able
to avoid copying those by making use of
`Orhpanage::referenceExternalData()`. However, there's no equivalent that
lets you reference whole message trees.
-Kenton
On Thu, Aug 11, 2022 at 9:03 AM Gyorgy Miru <g...@taszk.io> wrote:
> Hi,
>
> I am implementing a publisher/subscriber RPC interface with a large volume
> of messages.
> The server receives the incoming published messages and dispatches them to
> the subscribers after filtering without modifying the original message
> content. The filtering is done on separate worker threads.
>
> My question is whether this can be done with zero copy? The worst case is
> marshalling the received message struct into a native struct than building
> a struct parameter for each subscriber. That would be 1 + N copies.
>
> I tried to keep the original MyStruct::Reader around (that is received as
> a parameter upon publishing) but my understanding and experience is that it
> only holds a weak reference to the underlying data which is released as
> soon as the original context goes out of scope. See the code below for
> clarity.
>
>
> ------------------------------------------------------------------------------------------------------------
> kj::Promise<void> server::Server::publish(PublishContext context) {
>
> Event::Reader event = context.getParams().getEvent();
> // Is there a way to make Event::Reader-s data outlive this scope?
> // Or a convenient way to make a deep copy?
> eventRouter_.insertEvent(event);
>
> return kj::READY_NOW;
> };
>
> ------------------------------------------------------------------------------------------------------------
>
> The other part of the question is if I manage to keep the reader and its
> data (or I just make a new Event::Builder) can I reuse those to craft
> multiple subscriber dispatch requests. After all, the +N copies is that I
> really want to avoid. If i do something like below, would that work and
> would that result in N copies?
>
>
> ------------------------------------------------------------------------------------------------------------
> Event::Builder/Reader event;
> for (auto client: subscribers) {
> auto req = client.dispatchRequest();
> req.setEvent(event);
> req.send();
> }
>
> ------------------------------------------------------------------------------------------------------------
>
> Thank you for your time and help,
> -Gym
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/6dc55379-205d-4acc-ad95-33aa0f577b90n%40googlegroups.com
> <https://groups.google.com/d/msgid/capnproto/6dc55379-205d-4acc-ad95-33aa0f577b90n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/CAJouXQnWWxs7hPKZhV5XbH1-9v2SCg_NuP09S1wYVX8G-kZqAQ%40mail.gmail.com.
