Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread Jon Skeet
On Dec 7, 11:45 am, nightwalker leo [EMAIL PROTECTED] wrote: when I try to parse an addressbook file which has 2^20 records of person , my program complains like this: libprotobuf WARNING D:\protobuf-2.0.2\src\google\protobuf\io \coded_stream.cc:459] Reading dangerously large protocol

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread David Anderson
Do you really need to have the entire file in memory at once? Reading 64M of addresses into memory seems like the wrong approach (I could be wrong of course, since I don't know what you're doing with them). If you need to do something with each entry individually, you could do chunked reads:

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread Kenton Varda
On Sun, Dec 7, 2008 at 3:45 AM, nightwalker leo [EMAIL PROTECTED]wrote: when I try to parse an addressbook file which has 2^20 records of person , my program complains like this: libprotobuf WARNING D:\protobuf-2.0.2\src\google\protobuf\io \coded_stream.cc:459] Reading dangerously large

Re: Quick Hacky text_mode Parse in Python

2008-12-08 Thread Kenton Varda
Hey Petar, isn't there a patch someone was trying to submit that implements text format parsing? (For real, not by wrapping protoc.) What's the status of that? On Mon, Dec 8, 2008 at 5:03 AM, Nicholas Reid [EMAIL PROTECTED] wrote: Hi All, Firstly, just wanted to thank Kenton and the Google

Re: Quick Hacky text_mode Parse in Python

2008-12-08 Thread Petar Petrov
On Mon, Dec 8, 2008 at 10:21 AM, Kenton Varda [EMAIL PROTECTED] wrote: Hey Petar, isn't there a patch someone was trying to submit that implements text format parsing? (For real, not by wrapping protoc.) What's the status of that? I'll review it today. Hopefully the author hasn't forgotten

Re: Slicing support in Python

2008-12-08 Thread Kenton Varda
On Sat, Dec 6, 2008 at 1:03 AM, Alek Storm [EMAIL PROTECTED] wrote: But it does give us a lot of cool functionality, like adding the same message to two parents, and (yes!) slicing support. I thought this was common practice in C++, but it's been quite a while since I've coded it. Nope, in

Re: Slicing support in Python

2008-12-08 Thread Alek Storm
On Mon, Dec 8, 2008 at 1:16 PM, Kenton Varda [EMAIL PROTECTED] wrote: On Sat, Dec 6, 2008 at 1:03 AM, Alek Storm [EMAIL PROTECTED] wrote: Is it really that useful to have ByteSize() cached for repeated fields? If it's not, we get everything I mentioned above for free. I'm genuinely not sure

Doxygen and protobuf

2008-12-08 Thread Scott Stafford
Hi - Has anyone attempted to use Doxygen with .proto files? We're considering the possibility of looking into extending Doxygen to support the .proto format, but haven't started the project. Thoughts? Scott --~--~-~--~~~---~--~~ You received this message

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread nightwalker leo
thanks,could you give me an example plz On 12月8日, 下午4时10分, Jon Skeet [EMAIL PROTECTED] wrote: On Dec 7, 11:45 am, nightwalker leo [EMAIL PROTECTED] wrote: when I try to parse an addressbook file which has 2^20 records of person , my program complains like this: libprotobuf WARNING