Thanks for the reply. That perspective is very helpful. I've been wanting a system like Cap'n Proto for synchronizing a variety of configuration data between systems. I was on the fence about just copying the config data back and forth to plain old C++ structs, but when I first read the tips and best practices section, the *first* bullet point made me think it would be great to just pass around the Reader and avoid the extra copying and name-mapping code. But since I have an existing code base, not everything is going to be read-only. Thus the need for both getFoo and setFoo. My structs have about a hundred names each already, so going back and forth isn't trivial in terms of hand-written code.
But I'll start with that, and when I can I'll look into what it would take to implement generated "C++ structs" and the conversion logic (for where benefits of convenience outweigh those of speed). For my project, I've also been considering FlatBuffers and ProtocolBuffers. The two main drawbacks to Cap'n Proto *for me* are this whole issue and also the enforcement of camelCase which forces name mismatches with existing code. I've also run into some rough weather with pycapnp. Still I believe the drawbacks of FlatBuffers and Protobufs are even more severe, so I'm still plowing ahead with Cap'n Proto. You have my heartfelt thanks for all this, by the way!! Topher On Monday, March 29, 2021 at 8:55:42 PM UTC-6 ken...@cloudflare.com wrote: > Hi Topher, > > Unfortunately, Cap'n Proto is not well suited to representing mutable data > structures in-memory. Generally, MessageReaders are read-only, and > MessageBuilders are write-only. While technically you can modify builders > over time, the memory allocation patterns this tends to lead to are not > ideal. This is an unavoidable consequence of the memory allocation approach > needed to support zero-copy serialization: we need to allocate memory in a > contiguous space in order to be able to write it all out at once, but this > means memory fragmentation is impossible to avoid when data changes shape > over time. > > This is covered a bit in the "Best practices" section of the docs here: > https://capnproto.org/cxx.html#tips-and-best-practices > > A feature I've long wanted to add to Cap'n Proto is support for generating > appropriate "plain old C++ structs" (POCS) that mirror the Cap'n Proto > structs, with the ability to copy from a MesasgeReader into a POCS, and > from POCS to a MessageBuailder. Using POCS would mean you have to perform > at least one copy (not zero-copy anymore), but may be more convenient in > some use cases, especially when the structure will be modified in-memory > many times. > > -Kenton > > On Mon, Mar 29, 2021 at 6:36 PM Topher Cawlfield <tcawl...@gmail.com> > wrote: > >> I'm new to Cap'nProto, but am trying to use it as a replacement for >> structs in an existing project. I'm needing to serialize/deserialize >> obviously, but also just access the data within various programs. I'm >> having a very hard time keeping a MessageBuilder, Builder, and Reader in a >> class. >> >> It would be nice if I could deserialize a CapnProto struct in one >> method/function, modify it in another, read it in a third, and serialize in >> a fourth. But I'm stuck turning a PackedFdMessadeReader into a >> MessageBuilder (I don't mind any few memcpy's needed), and also creating a >> Builder instance as a member variable. >> >> I'm attaching my closest attempt at this. But it fails to print out the >> original data value -- I get a null string -- and it fails to write the >> modified value out. >> >> Any suggestions? >> >> Topher >> >> mystruct.capnp: >> @0xed859a09d409be91; >> >> struct MyStruct { >> foo @0 :Text; >> } >> >> main.cpp: >> #include <string> >> #include <iostream> >> #include <capnp/message.h> >> #include <capnp/serialize-packed.h> >> #include "mystruct.capnp.h" >> >> class MyClass { >> private: >> capnp::MallocMessageBuilder m_message; // Or capnp::MessageBuilder >> *m_message maybe >> >> public: >> MyStruct::Builder m_ms; >> >> MyClass() : >> m_message () >> , m_ms (m_message.initRoot<MyStruct>()) // sometimes segfaults here but >> absolutely required >> { >> m_ms = m_message.initRoot<MyStruct>(); // sometimes helps >> } >> >> void deserializeFrom(std::string filename) { >> int fd = 0; >> ::capnp::PackedFdMessageReader msg(fd); >> auto reader = msg.getRoot<MyStruct>(); >> m_message.setRoot(reader); // This leads to future problems with m_ms >> } >> >> void modify() { >> m_ms.setFoo("oh joy"); >> } >> >> void serializeTo(std::string filename) { >> int fd = 1; >> writePackedMessageToFd(fd, m_message); >> } >> }; >> >> int main() { >> MyClass c; >> c.deserializeFrom("some.capnp"); >> std::cerr << "foo is " << c.m_ms.getFoo().cStr() << std::endl; // read >> things occasionally >> c.modify(); >> c.serializeTo("next.capnp"); >> } >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Cap'n Proto" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to capnproto+...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/capnproto/6ce2d01c-43d4-44a6-9d26-0b3147742f55n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/capnproto/6ce2d01c-43d4-44a6-9d26-0b3147742f55n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/c7af1d22-1e86-4286-960c-5b52c5605e18n%40googlegroups.com.
