Re: [capnproto] [Feature Request] Putting List() object at last and allow reuse of the MallocMessageBuilder
Hi Kenton,
Got it, it makes sense. So if I remove `labelString` data from the
definition, I could use truncate function to operate my detections @4
:List(Detection2d); object. Thanks!
On Tuesday, 6 December 2022 at 10:43:17 pm UTC+8 ken...@cloudflare.com
wrote:
> Hi Hui,
>
> Again sorry for the slow reply.
>
> In fact, the functionality you suggest does exist today, in the form of
> `capnp::Orphan::truncate()`, which can be used to resize a list
> (truncate *or* extend), doing so in-place if possible. If the list is at
> the end of the message, it's possible it can be extended in-place. To use
> it, you would do:
>
> ```
> auto orphan = reader.disownDetections();
> orphan.truncate(newSize);
> reader.adoptDetections(kj::mv(orphan));
> ```
>
> HOWEVER, there is a problem that might apply to your sample use case: Your
> `Detection2d` struct contains `labelString @1 :Text`, which is a pointer
> type. When you set this field to a string value, the string is allocated at
> the end of the message. This means your list is *not* at the end of the
> message anymore, so you are no longer able to resize the list to make it
> longer. To be able to extend your list, you will need the list to contain
> only primitive types, so that it stays at the end of the message.
>
> -Kenton
>
> On Sat, Nov 5, 2022 at 9:16 PM Hui min wrote:
>
>> Hi Cap'n Proto Team,
>>
>> Thanks for the amazing tool you have created so far.
>>
>> Understand that with the current design of arena style memory allocation,
>> we cannot simply reuse the message, and re-init List() object to send it
>> again. This will cause the message to grow every time we send (memory leak).
>>
>> However, sending variable length data is still pretty common practice. If
>> we have to reallocate a brand new heap for new message, it is quite
>> wasteful. In most cases however, the variable length list is just one. So I
>> have an idea.
>>
>> ```
>> struct Detection2d {
>>
>> labelIdx @0 :UInt32;
>> labelString @1 :Text;
>> xmin @2 :Float32;
>> xmax @3 :Float32;
>> ymin @4 :Float32;
>> ymax @5 :Float32;
>> confidence @6 :Float32;
>>
>> }
>>
>> struct Detections2d {
>>
>> header @0 :import "header.capnp".Header;
>>
>> imageData @1 :Data; # should be RGB
>>
>> width @2 :UInt32;
>> height @3 :UInt32;
>>
>>detections @4 :List(Detection2d);
>> }
>> ```
>>
>> In this case, I have put the variable length object at the vary last.
>> That means everything in front is fix length. Is there a way i could force
>> the Capnproto to discard the memory of the old List and create a new one
>> directly from its old memory location, with out leaking a chunk of memory
>> in the arena?
>>
>> If it is not currently possible, do you think it is a convenient function
>> to be added? Thanks!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Cap'n Proto" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to capnproto+...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/capnproto/c500ece7-1871-4015-ba78-f456365a8bc3n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/capnproto/c500ece7-1871-4015-ba78-f456365a8bc3n%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/f92d55a5-9382-4e07-9d4d-f8a6f5766771n%40googlegroups.com.
[capnproto] capnp::messageToFlatArray to pre-allocated buffer?
Hi CapnProto Team, I understand the API to obtain a flat buffer of the built message is capnp::messageToFlatArray. However, the current API forces you to save the buffer on a returned kj::Array words object. Is there a way I could pass in a pre-allocated buffer, and let CapnProto to use that for serialisation (which essentially is creating a header + concat all the segments?)? Currently, some of the API has to be implemented in this way, when pre-allocated buffer is present. This introduced an extra copy and serialisation time. https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/include/ecal/msg/capnproto/publisher.h#L147-L150 A good kind of API would be like this: https://github.com/eclipse-ecal/ecal/blob/master/ecal/core/include/ecal/msg/protobuf/publisher.h#L162 Thanks in advance! -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/70bbe8b2-c06d-495f-ac6d-6eedecbdd305n%40googlegroups.com.
[capnproto] [Feature Request] Putting List() object at last and allow reuse of the MallocMessageBuilder
Hi Cap'n Proto Team,
Thanks for the amazing tool you have created so far.
Understand that with the current design of arena style memory allocation,
we cannot simply reuse the message, and re-init List() object to send it
again. This will cause the message to grow every time we send (memory leak).
However, sending variable length data is still pretty common practice. If
we have to reallocate a brand new heap for new message, it is quite
wasteful. In most cases however, the variable length list is just one. So I
have an idea.
```
struct Detection2d {
labelIdx @0 :UInt32;
labelString @1 :Text;
xmin @2 :Float32;
xmax @3 :Float32;
ymin @4 :Float32;
ymax @5 :Float32;
confidence @6 :Float32;
}
struct Detections2d {
header @0 :import "header.capnp".Header;
imageData @1 :Data; # should be RGB
width @2 :UInt32;
height @3 :UInt32;
detections @4 :List(Detection2d);
}
```
In this case, I have put the variable length object at the vary last. That
means everything in front is fix length. Is there a way i could force the
Capnproto to discard the memory of the old List and create a new one
directly from its old memory location, with out leaking a chunk of memory
in the arena?
If it is not currently possible, do you think it is a convenient function
to be added? Thanks!
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/c500ece7-1871-4015-ba78-f456365a8bc3n%40googlegroups.com.
Re: [capnproto] Performance of iterating through List vs. raw Data
Another comment, I would also agree probably using raw data as the type is
more appropriate, with a metadata information to encode the endianess.
This would be similiar
to http://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/PointCloud2.html
On Thursday, 15 August 2019 at 3:25:09 pm UTC+8 Philipp Wissmann wrote:
> Hi Kenton
>
> Thanks a lot for this amazingly fast reply and the explanations.
>
> We might try the direct pointer approach and it's very useful to know this
> possibility but for now I think having the data in a Data member seems to
> be working.
>
>
> Cheers!
> Philipp
>
> On Tuesday, August 13, 2019 at 7:43:08 PM UTC+2, Kenton Varda wrote:
>
>> Hi Philipp,
>>
>> This is a bit of an unusual case, where I imagine you are working with
>> bulk vector data forming a 3D mesh or some such, and in order to hand if
>> off to the graphics card, you really need a direct pointer to the
>> underlying data and you need it to be in a specific layout.
>>
>> Using Data is this case might make sense.
>>
>> Alternatively, here's one trick I thought of:
>>
>> Say your Vector3f is defined as:
>>
>> struct Vector3f { x @0 :Float32; y @1 :Float32; z @2 :Float32; }
>>
>> Note that Cap'n Proto pads every struct to an 8-byte boundary, so there's
>> 4 bytes of padding at the end of this struct. If that doesn't work for you,
>> then I think you have no choice but to use Data. But if the padding is OK,
>> then here's a way you can get a direct pointer to the data:
>>
>> const kj::byte* getRawPointer(capnp::List::Reader list) {
>> if (list.size() == 0) {
>> return nullptr;
>> } else {
>> capnp::AnyStruct::Reader any(list[0]);
>> KJ_REQUIRE(any.getPointerSection().size() == 0);
>> KJ_REQUIRE(any.getDataSection().size() == 16);
>> return any.getDataSection().begin();
>> }
>> }
>>
>> Here, you're getting a direct pointer to the "data section" of the first
>> struct in the list. Structs in a struct list are always stored
>> contiguously, so you can extend this out to the size of the list. You do
>> have to verify that the structs have the expected size, since technically
>> struct sizes are allowed to change to support schema evolution (in this
>> case, you'll never be able to add fields to Vector3f, except maybe a `w @3
>> :Float32` which would use the remaining padding without changing the size).
>> All structs in a struct list have the same size, so if the first struct
>> looks good, then the whole list is good.
>>
>> Hope that helps.
>>
>> -Kenton
>>
>> On Tue, Aug 13, 2019 at 9:32 AM wrote:
>>
> Hi
>>>
>>> We used Cap'n Proto to serialize data in a shared memory environment and
>>> defined some message types as structs containining List, i.e.
>>>
>>> struct Message{
>>>points @0 List(Vector3f)
>>> };
>>>
>>> Where Vector3f is another struct containing either 3 floats or a
>>> List(Float32). However, extracting the Vector3f's from the List is not fast
>>> enough for our use cases as you basically need to use std::copy or
>>> std::transform on the List. We instead now replaced the definition by
>>>
>>> struct Message{
>>>points @0 Data;
>>> };
>>>
>>> and just read and copy the array of bytes. Unfortunately, this basically
>>> gets rid of the nice schema language of Cap'n Proto. So, what's the most
>>> efficient way to define and read a contiguous array of the same data type?
>>>
>>> Best,
>>> Philipp
>>>
>>> *This message is confidential and only for the use of its addressee.
>>> Email communications are not secure and therefore we do not accept
>>> responsibility for the confidentiality or unaltered contents of this
>>> message.*
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>>
>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to capn...@googlegroups.com.
>>
>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/6a847434-c48b-4b18-9252-737562301464%40googlegroups.com
>>>
>>>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/fd4b0e14-4282-47f1-812d-af4da6ad4dabn%40googlegroups.com.
Re: [capnproto] Performance of iterating through List vs. raw Data
I want to ask a related question, and found out this thread.
>From what I understand in the discussion, if I have
struct Vector3d {
x @0 :Float64;
y @1 :Float64;
z @2 :Float64;
}
and then
struct Point3Array {
data @0 :List(Vector3d);
}
Will i get a perfectly packed array of Vector3d in memory, since it is
already 8-byte aligned?
A follow up question, would there be any changes in the memory layout, if I
change the definition of Vector3d to be:
struct Vector3d {
pt @0 :List(Float64)
}
and always insert 3 elements to Vector3d? I guess this won't work as
expected, as the List() is inheriently a variable length entity.
On Thursday, 15 August 2019 at 3:25:09 pm UTC+8 Philipp Wissmann wrote:
> Hi Kenton
>
> Thanks a lot for this amazingly fast reply and the explanations.
>
> We might try the direct pointer approach and it's very useful to know this
> possibility but for now I think having the data in a Data member seems to
> be working.
>
>
> Cheers!
> Philipp
>
> On Tuesday, August 13, 2019 at 7:43:08 PM UTC+2, Kenton Varda wrote:
>
>> Hi Philipp,
>>
>> This is a bit of an unusual case, where I imagine you are working with
>> bulk vector data forming a 3D mesh or some such, and in order to hand if
>> off to the graphics card, you really need a direct pointer to the
>> underlying data and you need it to be in a specific layout.
>>
>> Using Data is this case might make sense.
>>
>> Alternatively, here's one trick I thought of:
>>
>> Say your Vector3f is defined as:
>>
>> struct Vector3f { x @0 :Float32; y @1 :Float32; z @2 :Float32; }
>>
>> Note that Cap'n Proto pads every struct to an 8-byte boundary, so there's
>> 4 bytes of padding at the end of this struct. If that doesn't work for you,
>> then I think you have no choice but to use Data. But if the padding is OK,
>> then here's a way you can get a direct pointer to the data:
>>
>> const kj::byte* getRawPointer(capnp::List::Reader list) {
>> if (list.size() == 0) {
>> return nullptr;
>> } else {
>> capnp::AnyStruct::Reader any(list[0]);
>> KJ_REQUIRE(any.getPointerSection().size() == 0);
>> KJ_REQUIRE(any.getDataSection().size() == 16);
>> return any.getDataSection().begin();
>> }
>> }
>>
>> Here, you're getting a direct pointer to the "data section" of the first
>> struct in the list. Structs in a struct list are always stored
>> contiguously, so you can extend this out to the size of the list. You do
>> have to verify that the structs have the expected size, since technically
>> struct sizes are allowed to change to support schema evolution (in this
>> case, you'll never be able to add fields to Vector3f, except maybe a `w @3
>> :Float32` which would use the remaining padding without changing the size).
>> All structs in a struct list have the same size, so if the first struct
>> looks good, then the whole list is good.
>>
>> Hope that helps.
>>
>> -Kenton
>>
>> On Tue, Aug 13, 2019 at 9:32 AM wrote:
>>
> Hi
>>>
>>> We used Cap'n Proto to serialize data in a shared memory environment and
>>> defined some message types as structs containining List, i.e.
>>>
>>> struct Message{
>>>points @0 List(Vector3f)
>>> };
>>>
>>> Where Vector3f is another struct containing either 3 floats or a
>>> List(Float32). However, extracting the Vector3f's from the List is not fast
>>> enough for our use cases as you basically need to use std::copy or
>>> std::transform on the List. We instead now replaced the definition by
>>>
>>> struct Message{
>>>points @0 Data;
>>> };
>>>
>>> and just read and copy the array of bytes. Unfortunately, this basically
>>> gets rid of the nice schema language of Cap'n Proto. So, what's the most
>>> efficient way to define and read a contiguous array of the same data type?
>>>
>>> Best,
>>> Philipp
>>>
>>> *This message is confidential and only for the use of its addressee.
>>> Email communications are not secure and therefore we do not accept
>>> responsibility for the confidentiality or unaltered contents of this
>>> message.*
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>>
>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to capn...@googlegroups.com.
>>
>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/6a847434-c48b-4b18-9252-737562301464%40googlegroups.com
>>>
>>>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
