Another big advantage of writing the size first is that you can potentially write parsers which skip over sub-messages quickly, and perhaps parse them lazily. Currently the Google-provided implementations do not implement this, but there are some third-party implementations that use lazy parsing.
On Thu, Aug 5, 2010 at 4:18 PM, Jason Hsueh <[email protected]> wrote: > Groups are primarily deprecated because it used to be the case that groups > could not be reused in other types. Now groups can be reused, but other > messages must use the length-delimited format, rather than the group format. > In theory, the fact that groups do not require a length prefix makes them > more attractive. In practice, you typically end up computing message sizes > during serialization anyway (see below), so there is no benefit to using > groups. Messages are preferred stylistically: with a group you define a > message type and a field at once. I'll admit there are applications where > the group format is useful, specifically to stream a serialization by > constructing the message on the fly. But internally, there doesn't seem to > be a lot of demand for this. > > There are a number of performance optimizations (in C++, anyway) that > depend on having the total size before the data is serialized. When writing > to a string, having the size allows you to preallocate the array, avoiding > multiple reallocs. For output to abstract streams, it is often the case that > a message can fit into the buffer space available in the output, in which > case the faster code path serializing to a flat array can be used. This also > applies to embedded messages, so even if the parent message doesn't fit, > messages farther down the structure can still use the faster path. These > turned out to be pretty significant gains. (Again, C++ only. I don't know if > there are similar benefits in Java.) > > On Thu, Aug 5, 2010 at 6:53 AM, Evan Jones <[email protected]> wrote: > >> On Aug 5, 2010, at 9:16 , Ralf wrote: >> >>> I might be mistaken, but didn't "groups" use this approach - use a >>> special tag to indicate the end of a message? As only tags are >>> checked, there is no need to escape any data. >>> >> >> Good point, I forgot about groups. They definitely do use that approach. >> Maybe one of the Googlers on this list will have a better idea about why >> groups are now deprecated in favour of nested messages. >> >> >> >> Anyway, I was referring more to the implementation. For example, we >>> could first serialize the message to a ByteArrayOutputStream, then >>> write the result and its size to the output. Obviously this approach >>> is much slower, but I was wondering if there were other similar >>> approaches. >> >> > > Yes, you could do something like this. If you have some way to efficiently > copy the bytes, it might be a win to use this approach and avoid computing > sizes altogether. > > >> >> That's true, and would work. The other option would be to use fixed width >> integers for the lengths, so then you could "reserve" space in the buffer, >> serialize the message, then go back and fill in the length field. This would >> be an incompatible change to the serialization format, however. > > > In fact, we used to use a similar approach where we would assume that sub > messages are small. The code would leave a gap for the (varint-encoded) size > and serialize the message, then go back and fill in the length. If the > assumption was wrong, the data would have to be shifted. Using cached byte > sizes turned out to be a win in most cases though, particularly in complex > message structures. > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<protobuf%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
