Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread Jon Skeet
On Dec 7, 11:45 am, nightwalker leo [EMAIL PROTECTED] wrote: when I try to parse an addressbook file which has 2^20 records of person , my program complains like this: libprotobuf WARNING D:\protobuf-2.0.2\src\google\protobuf\io \coded_stream.cc:459] Reading dangerously large protocol

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread David Anderson
Do you really need to have the entire file in memory at once? Reading 64M of addresses into memory seems like the wrong approach (I could be wrong of course, since I don't know what you're doing with them). If you need to do something with each entry individually, you could do chunked reads:

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread Kenton Varda
On Sun, Dec 7, 2008 at 3:45 AM, nightwalker leo [EMAIL PROTECTED]wrote: when I try to parse an addressbook file which has 2^20 records of person , my program complains like this: libprotobuf WARNING D:\protobuf-2.0.2\src\google\protobuf\io \coded_stream.cc:459] Reading dangerously large

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread nightwalker leo
thanks,could you give me an example plz On 12月8日, 下午4时10分, Jon Skeet [EMAIL PROTECTED] wrote: On Dec 7, 11:45 am, nightwalker leo [EMAIL PROTECTED] wrote: when I try to parse an addressbook file which has 2^20 records of person , my program complains like this: libprotobuf WARNING

how to parse a file with millions of records with protobuf

2008-12-07 Thread nightwalker leo
when I try to parse an addressbook file which has 2^20 records of person , my program complains like this: libprotobuf WARNING D:\protobuf-2.0.2\src\google\protobuf\io \coded_stream.cc:459] Reading dangerously large protocol message. If the message turns out to be larger than 67108864 bytes,