I know this post is fairly old, but I'm curious - are you aware of any serialization protocols that are more focused on catering mutable data?
On Friday, August 8, 2014 at 3:32:37 PM UTC-5, Kenton Varda wrote: > > Hi Wes, > > Yes, you can argue that certain use cases might require a copy where with > protobufs they could be designed differently and avoid that copy. > > In my experience, most real-world uses of protobufs construct the whole > message just before sending. Whether or not this construction makes sense > to call a "copy" depends on the use case, but in any case that's not the > copy that Cap'n Proto is claiming to avoid. > > If you are trying to store some in-memory state that changes over time, > and you want to occasionally dump that state to disk / network, and you > would normally have been happy keeping the state in a protobuf object, > then, yes, with Cap'n Proto you now probably need to keep the state in some > other structure and do an extra copy every time you dump it. However, this > copy will still be a lot faster than protobuf encoding, since it's just a > lot of loads and stores with few branches. > > Actually, though, if you're careful, you can in fact store your state in a > Cap'n Proto structure. You just have to understand how Cap'n Proto manages > memory. If the changes you're making to your state don't ever involve > removing whole objects from the message tree, then there's no problem. Or, > if you're willing to occasionally do a sort of "garbage collection" pass by > making a copy of the whole message into a new MessageBuilder, then that can > also solve the problem -- and may in fact turn out to be a lot more > efficient than Protobuf memory management. For instance, you could do this > "GC" pass every time the space occupied by the message doubles. This would > give you amortized performance that's likely better than relying on > malloc(). But you have to understand what you're doing, which is why we > don't usually recommend it. :) > > -Kenton > > -- > Sandstorm.io is crowdfunding! http://igg.me/at/sandstorm > > > On Thu, Aug 7, 2014 at 6:09 PM, Wes Peters <[email protected] > <javascript:>> wrote: > >> I've just finished reading your article comparing features of Cap'n Proto >> with protobufs, SBE, and flatbuffers. >> >> I was involved in a selection trial for a serialization protocol two >> years ago, and we went with the wrong too for all the "right" reasons: the >> boss had already made a decision that we should be using XML to exchange >> data. As you probably know, XML serialization in C++ is generally an >> abysmal mess; I managed to make the best of it by discovering the quite >> good XSD compiler. Eventually we switched from XML to Boost encoding for >> most uses, because the XML encoding is just way too slow, so now we have >> all the lovely overhead of describing messages in XSD without even getting >> the questionable benefits of XML. I wanted to use protobufs, but got >> shouted down because it wasn't XML. >> >> So, this experience has left with with a quandry for the next project. I >> get to pick this time, and I say screw XML. So I'm out looking for >> serializers *again.* >> >> One thing that leaped out to me in your chart: you point out that the >> "zero copy" serialization libraries don't allow you to use the >> protocol-compiler generated objects as mutable state. Doesn't this mean >> that they are in fact *not* zero-copy then? If you have to maintain the >> state in another object, then create an object from that for serialization, >> you have in fact copied the data, probably in a constructor. So what we've >> actually managed to do is complicate the copying process... or have I >> mis-read what you wrote? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Cap'n Proto" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> Visit this group at http://groups.google.com/group/capnproto. >> > > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. Visit this group at https://groups.google.com/group/capnproto.
