Thanks, that worked!
Terri
On Mon, May 24, 2010 at 4:46 PM, Kenton Varda <ken...@google.com> wrote:
> My guess is that you're using a single CodedInputStream to read all your
> input, repeatedly calling message.ParseFromCodedStream(). Instead, create a
> new CodedInputStream for each message. If you construct it on the stack,
> there is no significant overhead to doing this:
> while (true) {
> CodedInputStream stream(&input);
> // read one message, or break if at EOF
> }
>
> On Mon, May 24, 2010 at 12:21 PM, Terri <terri.k...@gmail.com> wrote:
>>
>> Hi,
>>
>> I've been struggling to figure out just exactly how to do the many
>> smaller messages approach. I've implemented this strategy, which is
>> working except for a byte limit problem:
>>
>>
>> http://groups.google.com/group/protobuf/browse_thread/thread/038cc4ad000b4265/95981da7e07ce197?hide_quotes=no
>>
>> I also raised the byte limit using SetTotalBytesLimit to maxint.
>>
>> I use a python program to read my data form disk and package it up
>> into messages that are roughly 110 bytes each. Then I pipe it to a C++
>> program that reads messages and crunches. But, I still have a problem
>> because the total number of bytes of all my smaller messages is
>> greater than maxint and the C++ fails to read when it hits the limit.
>>
>> I like the protobuf approach to passing data, I just need to remove
>> that limit.
>>
>> What can I do?
>>
>> Thanks,
>> Terri
>>
>> On May 17, 7:00 pm, Jason Hsueh <jas...@google.com> wrote:
>> > There is a default byte size limit of 64MB when parsing protocol buffers
>> > -
>> > if a message is larger than that, it will fail to parse. This can be
>> > configured if you really need to parse larger messages, but it is
>> > generally
>> > not recommended. Additionally, ByteSize() returns a 32-bit integer, so
>> > there's an implicit limit on the size of data that can be serialized.
>> >
>> > You can certainly use protocol buffers in large data sets, but it's not
>> > recommended to have your entire data set be represented by a single
>> > message.
>> > Instead, see if you can break it up into smaller messages.
>> >
>> >
>> >
>> > On Mon, May 17, 2010 at 1:05 PM, sanikumbh <saniku...@gmail.com> wrote:
>> > > I wanted to get some opinion on large data sets and protocol buffers.
>> > > Protocol Buffer project page by google says that for data > 1
>> > > megabytes, one should consider something different but they don’t
>> > > mention what would happen if one crosses this limit. Are there any
>> > > known failure modes when it comes to the large data sets?
>> > > What are your observations, recommendations from your experience on
>> > > this front?
>> >
>> > > --
>> > > You received this message because you are subscribed to the Google
>> > > Groups
>> > > "Protocol Buffers" group.
>> > > To post to this group, send email to proto...@googlegroups.com.
>> > > To unsubscribe from this group, send email to
>> > >
>> > > protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
>> > > .
>> > > For more options, visit this group at
>> > >http://groups.google.com/group/protobuf?hl=en.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Protocol Buffers" group.
>> > To post to this group, send email to proto...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > protobuf+unsubscr...@googlegroups.com.
>> > For more options, visit this group
>> > athttp://groups.google.com/group/protobuf?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To post to this group, send email to proto...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> protobuf+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/protobuf?hl=en.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
