I actually don't know how the format works, but that is certainly one
possibility.
On Wed, Jan 13, 2010 at 5:19 AM, Delip Rao wrote:
> Thanks folks, that was very useful. Right now I have sequence of
> messages since we're processing serially. RecordIO seems like a great
> idea. Is the "framing
Thanks folks, that was very useful. Right now I have sequence of
messages since we're processing serially. RecordIO seems like a great
idea. Is the "framing format" just multiple messages in a file with an
inverted index in the beginning?
- Delip
On Tue, Jan 12, 2010 at 2:35 PM, Kenton Varda wro
So to rephrase what I said: You should break up your message in multiple
pieces that you store / send one at a time. Usually very large messages are
actually lists of smaller messages, so instead of using one big repeated
field, store each message separately. When storing to a file, it's probabl
Dang it, I got my mailing lists mixed up and referred to some things we
haven't released open source. Sigh.
On Tue, Jan 12, 2010 at 11:28 AM, Kenton Varda wrote:
> But you should consider a design that doesn't require you to send enormous
> messages. Protocol buffers are not well-optimized for
But you should consider a design that doesn't require you to send enormous
messages. Protocol buffers are not well-optimized for this sort of use.
For data stored on disk, consider storing multiple records in a RecordIO
file. For data passed over Stubby, consider streaming it in multiple
pieces.
The limit applies to the data source from which a message is parsed. So if
you want to parse a serialization of Foo, it applies to Foo. But if you
parse a bunch of Bar messages one by one, and add them individually to Bar,
then the limit only applies to each individual Bar.
You can change the limi
