I think the idea is to break up very large data sets into smaller packets so they can be 'streamed'. When I think of something like seismic data, stream based event handling makes the most sense. Can the data points be processed individually somehow, or do you need access to all of them (in memory) at once? -bmadigan
On Oct 22, 6:54 pm, [EMAIL PROTECTED] wrote: > OK, that makes sense. Thanks for the quick reply. > > I work at a seismic earthquake data center. We're looking at using > protocol buffers as a means of internally moving around processed > chunks of data. Seems to work pretty well, as long as the chunks > aren't too large > (which is a problem one way or another). But working with ~5 million > data points > doesn't seem to be any problem. > > "some other container format.".... Not exactly sure what that would > look like. > > Thanks again. > -B > > On Oct 22, 3:26 pm, "Kenton Varda" <[EMAIL PROTECTED]> wrote: > > > The "Partial" serialize and parse routines actually do something completely > > unrelated: they allow the message to be missing required fields. So, that > > doesn't help you. > > I'm afraid protocol buffers are not designed for storing very large > > collections in a single message. Instead, you should be thinking about > > representing each item in a collection using a protocol message, but then > > using some other container format. Using protocol buffers here makes the > > container format simpler because it only needs to deal with raw strings > > rather than having to worry about structured data. > > > On Wed, Oct 22, 2008 at 10:19 AM, <[EMAIL PROTECTED]> wrote: > > > > Is there a general strategy/methodology for dealing with very large > > > collections so that they do not need to be > > > completely held in memory before serializing and de-serializing? I > > > see, for example, SerializePartialToOstream() > > > and ParsePartialFromIstream() but no documentation of how to use it. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~----------~----~----~----~------~----~------~--~---
