So to rephrase what I said: You should break up your message in multiple pieces that you store / send one at a time. Usually very large messages are actually lists of smaller messages, so instead of using one big repeated field, store each message separately. When storing to a file, it's probably advantageous to use a "framing" format that lets you store multiple "records" such that you can seek to any particular record quickly -- using a large repeated field doesn't provide this anyway, so you need something else (we have some code internally that we call RecordIO).
BTW, we would love to open source the libraries I mentioned, it's just a
matter of finding the time to get it done.
On Tue, Jan 12, 2010 at 11:29 AM, Kenton Varda <ken...@google.com> wrote:
> Dang it, I got my mailing lists mixed up and referred to some things we
> haven't released open source. Sigh.
>
>
> On Tue, Jan 12, 2010 at 11:28 AM, Kenton Varda <ken...@google.com> wrote:
>
>> But you should consider a design that doesn't require you to send enormous
>> messages. Protocol buffers are not well-optimized for this sort of use.
>> For data stored on disk, consider storing multiple records in a RecordIO
>> file. For data passed over Stubby, consider streaming it in multiple
>> pieces.
>>
>> On Tue, Jan 12, 2010 at 9:40 AM, Jason Hsueh <jas...@google.com> wrote:
>>
>>> The limit applies to the data source from which a message is parsed. So
>>> if you want to parse a serialization of Foo, it applies to Foo. But if you
>>> parse a bunch of Bar messages one by one, and add them individually to Bar,
>>> then the limit only applies to each individual Bar.
>>>
>>> You can change the limit in your code if you create your own
>>> CodedInputStream and call its SetTotalBytesLimit method in C++, or its Java
>>> equivalent setSizeLimit.
>>>
>>> On Tue, Jan 12, 2010 at 8:41 AM, Delip Rao <delip...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm trying to understand protobuf message size limits. Is the 64M
>>>> message limit fixed or can it be changed via some compile option? If I
>>>> have a message Foo defined as:
>>>>
>>>> message Foo {
>>>> repeated Bar bars = 1;
>>>> }
>>>>
>>>> Will the limit apply to Foo or just the individual Bars?
>>>>
>>>> Thanks,
>>>> Delip
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Protocol Buffers" group.
>>>> To post to this group, send email to proto...@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
>>>> .
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/protobuf?hl=en.
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Protocol Buffers" group.
>>> To post to this group, send email to proto...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/protobuf?hl=en.
>>>
>>>
>>
>
-- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
