Re: [protobuf] Message limit
Thanks folks, that was very useful. Right now I have sequence of
messages since we're processing serially. RecordIO seems like a great
idea. Is the framing format just multiple messages in a file with an
inverted index in the beginning?
- Delip
On Tue, Jan 12, 2010 at 2:35 PM, Kenton Varda ken...@google.com wrote:
So to rephrase what I said: You should break up your message in multiple
pieces that you store / send one at a time. Usually very large messages are
actually lists of smaller messages, so instead of using one big repeated
field, store each message separately. When storing to a file, it's probably
advantageous to use a framing format that lets you store multiple
records such that you can seek to any particular record quickly -- using a
large repeated field doesn't provide this anyway, so you need something else
(we have some code internally that we call RecordIO).
BTW, we would love to open source the libraries I mentioned, it's just a
matter of finding the time to get it done.
On Tue, Jan 12, 2010 at 11:29 AM, Kenton Varda ken...@google.com wrote:
Dang it, I got my mailing lists mixed up and referred to some things we
haven't released open source. Sigh.
On Tue, Jan 12, 2010 at 11:28 AM, Kenton Varda ken...@google.com wrote:
But you should consider a design that doesn't require you to send
enormous messages. Protocol buffers are not well-optimized for this sort of
use. For data stored on disk, consider storing multiple records in a
RecordIO file. For data passed over Stubby, consider streaming it in
multiple pieces.
On Tue, Jan 12, 2010 at 9:40 AM, Jason Hsueh jas...@google.com wrote:
The limit applies to the data source from which a message is parsed. So
if you want to parse a serialization of Foo, it applies to Foo. But if you
parse a bunch of Bar messages one by one, and add them individually to Bar,
then the limit only applies to each individual Bar.
You can change the limit in your code if you create your own
CodedInputStream and call its SetTotalBytesLimit method in C++, or its Java
equivalent setSizeLimit.
On Tue, Jan 12, 2010 at 8:41 AM, Delip Rao delip...@gmail.com wrote:
Hi,
I'm trying to understand protobuf message size limits. Is the 64M
message limit fixed or can it be changed via some compile option? If I
have a message Foo defined as:
message Foo {
repeated Bar bars = 1;
}
Will the limit apply to Foo or just the individual Bars?
Thanks,
Delip
--
You received this message because you are subscribed to the Google
Groups Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google
Groups Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Message limit
The limit applies to the data source from which a message is parsed. So if
you want to parse a serialization of Foo, it applies to Foo. But if you
parse a bunch of Bar messages one by one, and add them individually to Bar,
then the limit only applies to each individual Bar.
You can change the limit in your code if you create your own
CodedInputStream and call its SetTotalBytesLimit method in C++, or its Java
equivalent setSizeLimit.
On Tue, Jan 12, 2010 at 8:41 AM, Delip Rao delip...@gmail.com wrote:
Hi,
I'm trying to understand protobuf message size limits. Is the 64M
message limit fixed or can it be changed via some compile option? If I
have a message Foo defined as:
message Foo {
repeated Bar bars = 1;
}
Will the limit apply to Foo or just the individual Bars?
Thanks,
Delip
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Message limit
But you should consider a design that doesn't require you to send enormous
messages. Protocol buffers are not well-optimized for this sort of use.
For data stored on disk, consider storing multiple records in a RecordIO
file. For data passed over Stubby, consider streaming it in multiple
pieces.
On Tue, Jan 12, 2010 at 9:40 AM, Jason Hsueh jas...@google.com wrote:
The limit applies to the data source from which a message is parsed. So if
you want to parse a serialization of Foo, it applies to Foo. But if you
parse a bunch of Bar messages one by one, and add them individually to Bar,
then the limit only applies to each individual Bar.
You can change the limit in your code if you create your own
CodedInputStream and call its SetTotalBytesLimit method in C++, or its Java
equivalent setSizeLimit.
On Tue, Jan 12, 2010 at 8:41 AM, Delip Rao delip...@gmail.com wrote:
Hi,
I'm trying to understand protobuf message size limits. Is the 64M
message limit fixed or can it be changed via some compile option? If I
have a message Foo defined as:
message Foo {
repeated Bar bars = 1;
}
Will the limit apply to Foo or just the individual Bars?
Thanks,
Delip
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
