Re: [capnproto] Can I do better?
Hi Jitesh,
Yes, this is possible, but a bit harder.
Instead of using MallocMessageBuilder, you would need to define your own
subclass of capnp::MessageBuilder that returns the buffers you want to use.
You will also need to keep track of framing on your own -- that is, keep
track of the number of segments and their sizes. Normally Cap'n Proto will
write a prefix on the message called the "segment table" which specifies
the length of each segment. But, the size of that table depends on the
number of segments, which normally isn't known in advance, so you wouldn't
know how much space to leave at the start of your message for it. That
said, if you are sure your message will always fit in the first segment,
then the segment table is not that useful anyway.
Also, if you're manually managing segments, then on the receiving end
you'll need to use capnp::SegmentArrayMessageReader, or write a custom
MessageReader subclass.
-Kenton
On Sat, Feb 26, 2022 at 8:09 AM Jitesh Khandelwal
wrote:
> Thanks Kenton, this is working well. Now the allocation in
> messageToFlatArray is not happening.
>
> Now, is it possible to build the message in the buffer directly ?
> Currently, I'm building in arena and then calling writeMessage to write
> into the buffer.
>
> --
> auto builder = MallocMessageBuilder(*arena*);
> ...
> ...
> auto message = arrayPtr(reinterpret_cast(*buffer*), N);
>
>
> auto aos = kj::ArrayOutputStream(message);
>
>
> capnp::writeMessage(aos, builder);
>
> --
> On Saturday, 26 February 2022 at 07:07:44 UTC+5:30 ken...@cloudflare.com
> wrote:
>
>> Hi Jitesh,
>>
>> Yes, you can avoid the allocation as follows:
>>
>> 1. Construct a kj::ArrayOutputStream that uses your destination buffer as
>> the output.
>> 2. Use capnp::writeMessage() to write the message to that stream.
>>
>> -Kenton
>>
>> On Fri, Feb 25, 2022 at 12:31 PM Jitesh Khandelwal
>> wrote:
>>
>>> I am trying to implement a middle layer which serialises some data from
>>> a data structure into a buffer, which is owned by and whose contents are
>>> written on the wire by an outer layer.
>>>
>>> Currently, my code looks as follows
>>>
>>>
>>> constexpr size_t N = 2048;
>>> char arena_buffer[N]{0};
>>> auto arena = arrayPtr(reinterpret_cast(arena_buffer), N /
>>> sizeof(word));
>>>
>>> Encode (some_data_structure, arena, buffer)
>>> {
>>> auto builder = MallocMessageBuilder(arena);
>>>
>>> // Step 1. set fields from some_data_structure using builder into
>>> arena
>>>
>>>
>>> // Step 2. convert to flat array (this allocates another array)
>>> auto message = messageToFlatArray(builder);
>>> auto bytes = message.asBytes();
>>>
>>> // Step 3. copy to destination
>>> memcpy(buffer, bytes.begin(), bytes.size());
>>> }
>>>
>>>
>>>
>>> My question is, Can I do better?
>>>
>>> My understanding is that the extra allocation in Step 2, is because of
>>> the segment framing protocol ?
>>>
>>> Can I bypass it somehow, as I am sure that the arena is big enough for
>>> my message and there'll be only 1 segment ?
>>>
>>> And if yes, then may be I can use the buffer as the arena, as in build
>>> directly into the output buffer?
>>>
>>>
>>>
>>> Thanks,
>>> Jitesh
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/a6378e3d-fd2e-4009-8cf4-595011170dc9n%40googlegroups.com
>>>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/b09290d5-e2d3-4711-b309-9bcdaaade8e5n%40googlegroups.com
>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
Re: [capnproto] Can I do better?
Thanks Kenton, this is working well. Now the allocation in
messageToFlatArray is not happening.
Now, is it possible to build the message in the buffer directly ?
Currently, I'm building in arena and then calling writeMessage to write
into the buffer.
--
auto builder = MallocMessageBuilder(*arena*);
...
...
auto message = arrayPtr(reinterpret_cast(*buffer*), N);
auto aos = kj::ArrayOutputStream(message);
capnp::writeMessage(aos, builder);
--
On Saturday, 26 February 2022 at 07:07:44 UTC+5:30 ken...@cloudflare.com
wrote:
> Hi Jitesh,
>
> Yes, you can avoid the allocation as follows:
>
> 1. Construct a kj::ArrayOutputStream that uses your destination buffer as
> the output.
> 2. Use capnp::writeMessage() to write the message to that stream.
>
> -Kenton
>
> On Fri, Feb 25, 2022 at 12:31 PM Jitesh Khandelwal
> wrote:
>
>> I am trying to implement a middle layer which serialises some data from a
>> data structure into a buffer, which is owned by and whose contents are
>> written on the wire by an outer layer.
>>
>> Currently, my code looks as follows
>>
>>
>> constexpr size_t N = 2048;
>> char arena_buffer[N]{0};
>> auto arena = arrayPtr(reinterpret_cast(arena_buffer), N /
>> sizeof(word));
>>
>> Encode (some_data_structure, arena, buffer)
>> {
>> auto builder = MallocMessageBuilder(arena);
>>
>> // Step 1. set fields from some_data_structure using builder into
>> arena
>>
>>
>> // Step 2. convert to flat array (this allocates another array)
>> auto message = messageToFlatArray(builder);
>> auto bytes = message.asBytes();
>>
>> // Step 3. copy to destination
>> memcpy(buffer, bytes.begin(), bytes.size());
>> }
>>
>>
>>
>> My question is, Can I do better?
>>
>> My understanding is that the extra allocation in Step 2, is because of
>> the segment framing protocol ?
>>
>> Can I bypass it somehow, as I am sure that the arena is big enough for my
>> message and there'll be only 1 segment ?
>>
>> And if yes, then may be I can use the buffer as the arena, as in build
>> directly into the output buffer?
>>
>>
>>
>> Thanks,
>> Jitesh
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Cap'n Proto" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to capnproto+...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/capnproto/a6378e3d-fd2e-4009-8cf4-595011170dc9n%40googlegroups.com
>>
>>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/b09290d5-e2d3-4711-b309-9bcdaaade8e5n%40googlegroups.com.
Re: [capnproto] Can I do better?
Hi Jitesh,
Yes, you can avoid the allocation as follows:
1. Construct a kj::ArrayOutputStream that uses your destination buffer as
the output.
2. Use capnp::writeMessage() to write the message to that stream.
-Kenton
On Fri, Feb 25, 2022 at 12:31 PM Jitesh Khandelwal
wrote:
> I am trying to implement a middle layer which serialises some data from a
> data structure into a buffer, which is owned by and whose contents are
> written on the wire by an outer layer.
>
> Currently, my code looks as follows
>
>
> constexpr size_t N = 2048;
> char arena_buffer[N]{0};
> auto arena = arrayPtr(reinterpret_cast(arena_buffer), N /
> sizeof(word));
>
> Encode (some_data_structure, arena, buffer)
> {
> auto builder = MallocMessageBuilder(arena);
>
> // Step 1. set fields from some_data_structure using builder into arena
>
>
> // Step 2. convert to flat array (this allocates another array)
> auto message = messageToFlatArray(builder);
> auto bytes = message.asBytes();
>
> // Step 3. copy to destination
> memcpy(buffer, bytes.begin(), bytes.size());
> }
>
>
>
> My question is, Can I do better?
>
> My understanding is that the extra allocation in Step 2, is because of the
> segment framing protocol ?
>
> Can I bypass it somehow, as I am sure that the arena is big enough for my
> message and there'll be only 1 segment ?
>
> And if yes, then may be I can use the buffer as the arena, as in build
> directly into the output buffer?
>
>
>
> Thanks,
> Jitesh
>
> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/a6378e3d-fd2e-4009-8cf4-595011170dc9n%40googlegroups.com
>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/CAJouXQkanKYk%3D1dgdB%2BkDFc7xR%3DQ4o5d%3DoW1o4kBk7RaBu5__g%40mail.gmail.com.
[capnproto] Can I do better?
I am trying to implement a middle layer which serialises some data from a
data structure into a buffer, which is owned by and whose contents are
written on the wire by an outer layer.
Currently, my code looks as follows
constexpr size_t N = 2048;
char arena_buffer[N]{0};
auto arena = arrayPtr(reinterpret_cast(arena_buffer), N /
sizeof(word));
Encode (some_data_structure, arena, buffer)
{
auto builder = MallocMessageBuilder(arena);
// Step 1. set fields from some_data_structure using builder into arena
// Step 2. convert to flat array (this allocates another array)
auto message = messageToFlatArray(builder);
auto bytes = message.asBytes();
// Step 3. copy to destination
memcpy(buffer, bytes.begin(), bytes.size());
}
My question is, Can I do better?
My understanding is that the extra allocation in Step 2, is because of the
segment framing protocol ?
Can I bypass it somehow, as I am sure that the arena is big enough for my
message and there'll be only 1 segment ?
And if yes, then may be I can use the buffer as the arena, as in build
directly into the output buffer?
Thanks,
Jitesh
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/a6378e3d-fd2e-4009-8cf4-595011170dc9n%40googlegroups.com.
