Re: [capnproto] capnp::messageToFlatArray to pre-allocated buffer?

['Kenton Varda' via Cap'n Proto]

Hi Hui,

Sorry for the slow reply, I was on leave.

messageToFlatArray() is a convenience function for when you don't mind
allocation and copying. If you want more control over serialization, you
can use MessageBuilder::getSegmentsForOutput() to get direct pointers to
the message's underlying buffer space, without copying anything. You can
then serialize these segments as you see fit. Note that you are responsible
for tracking the segment boundaries in this case. On the receiving end,
pass the same array-of-arrays into SegmentArrayMessageReader to parse the
message. Again, this doesn't perform any allocation or copies; that's up to
you to manage.

Another option is to implement your own kj::OutputStream, and then use
capnp::writeMessage() to write a message to the stream. This will always
write the entire message using one single call to write() on the underlying
stream, passing an array-of-arrays that point into the original message
buffer -- again, no copies. You could then copy into your scratch buffer or
do whatever else you want. An advantage of this approach (using
OutputStream instead of getSegmentsForOutput()) is that writeMessage() will
also build a segment table for you and include that in the output, so on
the receiving end you can use FlatArrayMessageReader instead of
SegmentArrayMessageReader.

-Kenton

On Sat, Nov 5, 2022 at 9:29 PM Hui min  wrote:

> Hi CapnProto Team,
>
> I understand the API to obtain a flat buffer of the built message is
> capnp::messageToFlatArray.
>
> However, the current API forces you to save the buffer on a returned
> kj::Array words object. Is there a way I could pass in a
> pre-allocated buffer, and let CapnProto to use that for serialisation
> (which essentially is creating a header + concat all the segments?)?
>
> Currently, some of the API has to be implemented in this way, when
> pre-allocated buffer is present. This introduced an extra copy and
> serialisation time.
>
> https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/include/ecal/msg/capnproto/publisher.h#L147-L150
>
> A good kind of API would be like this:
>
> https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/include/ecal/msg/protobuf/publisher.h#L162
>
> Thanks in advance!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/70bbe8b2-c06d-495f-ac6d-6eedecbdd305n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/CAJouXQkZ3uQPfobVzqZnOWsn6m-AhW3kt-tGZxAoFTvg_nK7vw%40mail.gmail.com.

[capnproto] capnp::messageToFlatArray to pre-allocated buffer?

[Hui min]

Hi CapnProto Team,

I understand the API to obtain a flat buffer of the built message is 
capnp::messageToFlatArray.

However, the current API forces you to save the buffer on a returned 
kj::Array words object. Is there a way I could pass in a 
pre-allocated buffer, and let CapnProto to use that for serialisation 
(which essentially is creating a header + concat all the segments?)?

Currently, some of the API has to be implemented in this way, when 
pre-allocated buffer is present. This introduced an extra copy and 
serialisation time.
https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/include/ecal/msg/capnproto/publisher.h#L147-L150

A good kind of API would be like this:
https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/include/ecal/msg/protobuf/publisher.h#L162

Thanks in advance!

-- 
You received this message because you are subscribed to the Google Groups 
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/capnproto/70bbe8b2-c06d-495f-ac6d-6eedecbdd305n%40googlegroups.com.