Thanks, I've started to understand this better. Indeed, I have to implement my own approach for I/O - protobuf alone is not enough. I only worry that my own I/O to read/write records will not be cross platform, so I could not benefit from the strength of this package.
On Oct 8, 1:24 pm, Kenton Varda <ken...@google.com> wrote: > On Thu, Oct 8, 2009 at 10:57 AM, sergei175 <sergei...@googlemail.com> wrote: > > 1) After event 500, even 200MB memory is not enough. > > 2) It's slower by factor ~5 compare to the java serialization with > > the > > compression. > > Protocol Buffers do not include compression, so to make this comparison fair > you would need to add compression on top of them too. If your speed is > dominated by file I/O time (likely!) then you might find that this makes > protocol buffers faster. > > > 3) File size is very large. I do not know how to fill > > compressed recorsd on fly using this package. > > Use java.util.zip.GZIPOutputStream. > > > Finally, there is no even sensible approach to append new "Records" > > to the existing file (without "merge", which in fact has to parse > > the > > existing file first!) > > http://code.google.com/apis/protocolbuffers/docs/techniques.html#stre... > > Protocol Buffers convert between raw bytes and structures. They are not > intended to provide a mechanism for managing multiple individually-loadable > records. If you have a very large data set, you need to split that set into > individual records in order to avoid reading/writing the whole thing at > once. Each individual record can be encoded using protobufs, but you should > not encode the entire file as a protobuf. > > > So, I do not see any superiority of Protocol Buffers compare > > to use file formats, it's actually much worst as it come to such > > situations.. > > By all means, don't use them then. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~----------~----~----~----~------~----~------~--~---
