To be clear, I do not encode the entire file! Each file contains many small messages, each of which is stored as a length delimited record. It is just that there are quite a few messages bundled in one file.
I'm assuming that Evan's assessment still stand? Cheers, Nader On 6/3/2010 15:05 Kenton Varda writes: > Note that writing a 100GB file using CodedStream is probably a bad idea > because: > - Readers will have to read the entire file sequentially; they will not be > able to seek to particular parts. > - One bit of corruption anywhere in the file could potentially render the > entire rest of the file unreadable. > > Remember that this stuff was designed for small messages. You should really > use some sort of seekable, fault-tolerant container format for 100GB of > data. You can still encode each individual message using protobufs, which > is useful as it allows the container format to treat each message as a > simple byte blob. > > On Thu, Jun 3, 2010 at 12:43 PM, Evan Jones <ev...@mit.edu> wrote: > > > On Jun 3, 2010, at 15:29 , Nader Salehi wrote: > > > >> It is not a single object; I am writing into a coded output stream > >> file which could grow to much larger than 2GB (it's more like 100GB). > >> I also have to read from this file. > >> > >> Is there a performance hit in the above-mentioned scenario? > >> > > > > No, this should work just fine. On the input size, you'll need to call > > CodedInputStream.resetSizeCounter() after each message, otherwise you'll run > > into the size limit. > > > > > > Evan > > > > -- > > Evan Jones > > http://evanjones.ca/ > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Protocol Buffers" group. > > To post to this group, send email to proto...@googlegroups.com. > > To unsubscribe from this group, send email to > > protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com> > > . > > For more options, visit this group at > > http://groups.google.com/group/protobuf?hl=en. > > > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
