Sorry if this has been covered before. I searched but couldn't find a
complete answer (or at least what I thought was complete). I'll give a
little background and some small example code as I believe it will help.
I'm using C++ and optimizing my protocol buffer messages for LITE_RUNTIME.
I have a class called Resource and it currently looks something like this
class Resource
{
public:
// other stuff that isn't necessary for this example
::google::protobuf::MessageLite* createNewProtoBuff() const;
// More stuff
};
As you can see the Resource class has a function which creates a new
protocol buffer "message" in memory.
I also have a class called ResourceMap which serves as a collection of
Resources. Since there can be any number of Resources in a ResourceMap and
I don't want to go over the 1 meg "limit" (not a hard limit I know) for a
single message, when I serialize out to a file for a ResourceMap, I first
serialize out a "header" protocol buffer message to the file and then
serialize each Resource's protocol buffer message.
The header looks like this:
message ResourceMapHeaderPB {
message ResourceEntryPB
{
optional string resource_name = 1;
optional uint32 resource_byte_offset_from_header = 2; // Used if we
want to deserialize a single Resource in a ResourceMap an not all
optional uint32 resource_size_bytes = 3; // Used to set
CodedInputStream limit upon deserialization
}
repeated ResourceEntryPB resource_entry = 1;
}
The serialization to file code for ResourceMap looks like this:
// Loop over all Resources in the ResourceMap, creating MessageLite* from
the resources
for ( iter = allResources.begin(); iter != allResources.end(); ++iter )
{
// Create new entry in the header
entryPB = headerPB->add_resource_entry();
entryPB->set_resource_name(iter->Name);
entryPB->set_resource_size_bytes(resourcePB->ByteSize());
entryPB->set_resource_byte_offset_from_header(currentByteOffsetFromHeader);
currentByteOffsetFromHeader += resourcePB->ByteSize();
}
// Once all MessageLite* are constructed, write header and messages to
CodedOutputStream
codedOut->WriteLittleEndian32(headerPB->ByteSize());
headerPB->SerializeToCodedStream(codedOut);
for ( auto iter = resourcePBs.begin(); resourcePBs.end() != iter; ++iter )
{
(*iter)->SerializeToCodedStream(codedOut);
}
As you can see, the header needs to know the size of the Message in bytes
to support a feature (deserializing a single resource instead of all in a
given file).
Right now, the code works fine and is great. But, there is one problem, *it
is possible for the Messages created from a Resource to be > 1 MB*.
In this scenario, where I don't want a single MessageLite to be > 1 MB,
what can I do?
I was thinking of making a ResourceSerialization class that would be
something like
class ResourceSerialization
{
public:
uint32 GetSerializationSizeBytes();
void SerializeToCodedStream(CodedOutputStream* codedOut);
};
For Resources which could serialize to a single message under 1 MB, the
size function above could just return the size of a wrapped MessageLite. In
other scenarios where the ResourceSerialization might consist of writing
out the size of a message, then a message, then size of the next message,
then the next message, etc. etc. (as indicated in the google protocol
buffer documentation under "techniques") how would I find the "total" size
of what would be serialized? Is there a way to know the size that would be
written by CodedOutputStream::WriteVarint32(uint32 value) with a particular
value?
Sorry if this is all convoluted and confusing. I can clarify as needed.
Thanks ahead of time!
-Jonathan
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.
