Do you really need to have the entire file in memory at once? Reading
>64M of addresses into memory seems like the wrong approach (I could
be wrong of course, since I don't know what you're doing with them).

If you need to do something with each entry individually, you could do
chunked reads: when writing, instead of serializing the whole message
at once, build several messages of ~10M each, and write them out with
a length prefix. When reading back, use the length prefix to yield
nicely sized chunks of your address book. There may even be a nice way
to do this implicitely at the input/output stream level, if it is
aware of field boundaries, but I don't have a good enough handle on
the implementation to say.

If you need to find a specific entry in the address book, you should
sort the address book. You then chunk in the same manner, and add an
index message at the end of the file that lists the start offset of
all chunks. You can then do binary search over the chunks (even more
efficiently if the index includes the start and end "keys" of your
chunks, e.g. last names) to locate the chunk you want.

If none of these answers are satisfactory, and you really need the
entire multi-hundred-megabyte message loaded at once, I guess you can
use SetTotalBytesLimit() to raise the safety limits to whatever you
feel is necessary. But usually, when I try to load a bunch of small
messages as one monolithic block, I find that my data format isn't
adapted to what I want to do.

Hope this helps a little
- Dave

On Sun, Dec 7, 2008 at 12:45 PM, nightwalker leo <[EMAIL PROTECTED]> wrote:
> when I try to parse an addressbook file which has 2^20 records of
> person , my program complains like this:
> libprotobuf WARNING D:\protobuf-2.0.2\src\google\protobuf\io
> \] Reading dangerously large protocol message.  If
> the message turns out to be larger than 67108864 bytes, parsing will
> be halted for security reasons.  To increase the limit (or to disable
> these warnings), see CodedInputStream::SetTotalBytesLimit() in google/
> protobuf/io/coded_stream.h.
> how to deal with the problem in an elegant way instead of increasing
> the limit or simply turning off the warning message?
> >

You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at

Reply via email to