Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread nightwalker leo
thank you, I really appreciate your work On 12月9日, 上午2时17分, Kenton Varda <[EMAIL PROTECTED]> wrote: > On Sun, Dec 7, 2008 at 3:45 AM, nightwalker leo <[EMAIL PROTECTED]>wrote: > > > > > when I try to parse an addressbook file which has 2^20 records of > > person , my program complains like this:

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread nightwalker leo
eh, it seems that i misunderstood the design of protobuf, thank you so much, good luck & best wishes On 12月8日, 下午8时00分, "David Anderson" <[EMAIL PROTECTED]> wrote: > Do you really need to have the entire file in memory at once? Reading>64M of > addresses into memory seems like the wrong approach

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread nightwalker leo
thanks,could you give me an example plz On 12月8日, 下午4时10分, Jon Skeet <[EMAIL PROTECTED]> wrote: > On Dec 7, 11:45 am, nightwalker leo <[EMAIL PROTECTED]> wrote: > > > when I try to parse an addressbook file which has 2^20 records of > > person , my program complains like this: > > libprotobuf WAR

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread Kenton Varda
On Sun, Dec 7, 2008 at 3:45 AM, nightwalker leo <[EMAIL PROTECTED]>wrote: > > when I try to parse an addressbook file which has 2^20 records of > person , my program complains like this: > libprotobuf WARNING D:\protobuf-2.0.2\src\google\protobuf\io > \coded_stream.cc:459] Reading dangerously larg

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread David Anderson
Do you really need to have the entire file in memory at once? Reading >64M of addresses into memory seems like the wrong approach (I could be wrong of course, since I don't know what you're doing with them). If you need to do something with each entry individually, you could do chunked reads: whe

Re: how to parse a file with millions of records with protobuf

2008-12-08 Thread Jon Skeet
On Dec 7, 11:45 am, nightwalker leo <[EMAIL PROTECTED]> wrote: > when I try to parse an addressbook file which has 2^20 records of > person , my program complains like this: > libprotobuf WARNING D:\protobuf-2.0.2\src\google\protobuf\io > \coded_stream.cc:459] Reading dangerously large protocol me

how to parse a file with millions of records with protobuf

2008-12-07 Thread nightwalker leo
when I try to parse an addressbook file which has 2^20 records of person , my program complains like this: libprotobuf WARNING D:\protobuf-2.0.2\src\google\protobuf\io \coded_stream.cc:459] Reading dangerously large protocol message. If the message turns out to be larger than 67108864 bytes, pars