On Tue, Feb 8, 2011 at 11:23 AM, Evan Jones <[email protected]> wrote:
> Sorry, just an example of why you might want a different protocol. If I've
> streamed 10e9 messages to disk, I don't want this stream to break if there
> is some weird corruption in the middle, so I want some protocol that can
> "resume" from corruption.
>
Ah, yes. This isn't an appropriate protocol for enormous files. It's
more targeted at network protocols.
Although, you might be able to build a decent seekable file protocol on top
of it, by choosing a random string to use as a sync point, then writing that
string every now and then...
message FileStream {
repeated string sync_point = 1;
repeated Foo foo = 2;
repeated Bar bar = 3;
...
}
When writing, after every few messages, write a copy of sync_point. Then,
you can seek to an arbitrary position in the file by looking for a nearby
copy of the sync point byte sequence, and starting to parse immediately
after that. The sync point just needs to be an 128-bit (or so)
cryptographically random sequence, chosen differently for each file, so that
there's no chance that the bytes will appear in the file by accident.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
