On Fri, Jan 3, 2014 at 12:59 PM, <[email protected]> wrote: > Sorry if this has been covered before. I searched but couldn't find a > complete answer (or at least what I thought was complete). I'll give a > little background and some small example code as I believe it will help. > I'm using C++ and optimizing my protocol buffer messages for LITE_RUNTIME. > > I have a class called Resource and it currently looks something like this > > class Resource > { > public: > // other stuff that isn't necessary for this example > ::google::protobuf::MessageLite* createNewProtoBuff() const; > > // More stuff > }; > > As you can see the Resource class has a function which creates a new > protocol buffer "message" in memory. > > I also have a class called ResourceMap which serves as a collection of > Resources. Since there can be any number of Resources in a ResourceMap and > I don't want to go over the 1 meg "limit" (not a hard limit I know) for a > single message, when I serialize out to a file for a ResourceMap, I first > serialize out a "header" protocol buffer message to the file and then > serialize each Resource's protocol buffer message. > > The header looks like this: > > message ResourceMapHeaderPB { > > message ResourceEntryPB > > { > > optional string resource_name = 1; > > optional uint32 resource_byte_offset_from_header = 2; // Used if we > want to deserialize a single Resource in a ResourceMap an not all > > optional uint32 resource_size_bytes = 3; // Used to set > CodedInputStream limit upon deserialization > > } > > repeated ResourceEntryPB resource_entry = 1; > > } > > The serialization to file code for ResourceMap looks like this: > > > // Loop over all Resources in the ResourceMap, creating MessageLite* from > the resources > for ( iter = allResources.begin(); iter != allResources.end(); ++iter ) > { > // Create new entry in the header > > entryPB = headerPB->add_resource_entry(); > > entryPB->set_resource_name(iter->Name); > > entryPB->set_resource_size_bytes(resourcePB->ByteSize()); > > entryPB->set_resource_byte_offset_from_header > (currentByteOffsetFromHeader); > > currentByteOffsetFromHeader += resourcePB->ByteSize(); > > } > > // Once all MessageLite* are constructed, write header and messages to > CodedOutputStream > > codedOut->WriteLittleEndian32(headerPB->ByteSize()); > > headerPB->SerializeToCodedStream(codedOut); > > for ( auto iter = resourcePBs.begin(); resourcePBs.end() != iter; ++iter ) > > { > > (*iter)->SerializeToCodedStream(codedOut); > > } > As you can see, the header needs to know the size of the Message in bytes > to support a feature (deserializing a single resource instead of all in a > given file). > > Right now, the code works fine and is great. But, there is one problem, *it > is possible for the Messages created from a Resource to be > 1 MB*. > > In this scenario, where I don't want a single MessageLite to be > 1 MB, > what can I do? > > I was thinking of making a ResourceSerialization class that would be > something like > > class ResourceSerialization > { > public: > > uint32 GetSerializationSizeBytes(); > > void SerializeToCodedStream(CodedOutputStream* codedOut); > > }; > > For Resources which could serialize to a single message under 1 MB, the > size function above could just return the size of a wrapped MessageLite. In > other scenarios where the ResourceSerialization might consist of writing > out the size of a message, then a message, then size of the next message, > then the next message, etc. etc. (as indicated in the google protocol > buffer documentation under "techniques") how would I find the "total" size > of what would be serialized? Is there a way to know the size that would be > written by CodedOutputStream::WriteVarint32(uint32 value) with a > particular value? > CodedOutputStream::VarintSize32(uint32 value)?
> > Sorry if this is all convoluted and confusing. I can clarify as needed. > Thanks ahead of time! > > -Jonathan > > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/groups/opt_out.
