Similarly to what Jason suggested, you may have summary blocks which lets you quick inspect value ranges. Like:
message Record { optional int id = 1; // fields only you know about. } message RecordBlock { repeated Record record = 1; // You could declare these as 'bytes' type // instead of 'Record' type. This avoids parsing // each Record if you're not going to use it. optional int max_id = 2; optional int min_id = 3; } message RecordsFile { repeated RecordBlock raw_blocks = 1; // Readly available RecordBlock. repeated bytes gziped_blocks = 2; // Gzip encoded RecordBlock. repeated bytes other_encoding_blocks = 3; // Other encoded RecordBlock. } You may use RecordsFile to append records to your file. For parsing, you can use CodedInputStream. On Tue, Apr 3, 2012 at 15:26, Jason Hsueh <jas...@google.com> wrote: > > In general protobuf doesn't try to provide these kinds of container formats. > Gzip*Stream is meant to deal with data that is valid gzip; inserting frames > to make it searchable would break the format. In the archives you may find > other references to the internal container file format that Google has; this > effectively has blocks of individual records, and it's possible to skip > blocks or seek to a position (or the next valid record after a position). It > sounds like you want something similar, with your events as the records, and > each block as the 'key frames'. Rather than compressing the entire file, you > could perform block level compression, and treat each individual record as a > separate stream that can be decoded into your protobuf. > > > On Mon, Apr 2, 2012 at 5:30 AM, Roman Vinogradov <vinogradov.ro...@gmail.com> > wrote: >> >> Hello, >> >> I use protobuf to store events in playback file for further replaying. >> I use Coded*Stream on top of Gzip*Stream which is on top of >> File*Stream. The playback file is big enough, can be up to 3 GB a day >> (hundreds of millions of events). >> Everything works ok - all events encoded in the file can be replayed >> back just fine. >> Now I need to navigate through the playback (forth and back), kind of >> rewinding. >> I wanted to periodically insert special key frames that include such >> information as byte offset of previous key frame (to rewind back) and >> similar offset of the next key frame (to rewind forth), plus some >> other key data. So I would navigate through the file faster by >> skipping big portions of events between such key frames. >> In order to maintain key frames linked each to other I need to move >> back in the output buffer, modify previous key frame with correct >> value of offset field and move forward to the initial position in >> order to serialize next key frame and further events and so on. If >> needed key frame can be of fixed size (e.g. sfixed64 types can be >> used). >> Reading and partial parsing of all events until good one is found is >> not an option. >> >> How is this possible with protobuf implementation? e.g. >> CodedOutputStream::Skip(int count) doesn't accept negative count as a >> parameter. >> >> Perhaps there is alternate way to reach the goal - fast navigation in >> the binary gzip'ed file that includes messages of variable length. >> Any thoughts? >> >> Thank you in advance. >> >> /Roman >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To post to this group, send email to protobuf@googlegroups.com. >> To unsubscribe from this group, send email to >> protobuf+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/protobuf?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to protobuf@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.