In general protobuf doesn't try to provide these kinds of container formats. Gzip*Stream is meant to deal with data that is valid gzip; inserting frames to make it searchable would break the format. In the archives you may find other references to the internal container file format that Google has; this effectively has blocks of individual records, and it's possible to skip blocks or seek to a position (or the next valid record after a position). It sounds like you want something similar, with your events as the records, and each block as the 'key frames'. Rather than compressing the entire file, you could perform block level compression, and treat each individual record as a separate stream that can be decoded into your protobuf.
On Mon, Apr 2, 2012 at 5:30 AM, Roman Vinogradov <[email protected] > wrote: > Hello, > > I use protobuf to store events in playback file for further replaying. > I use Coded*Stream on top of Gzip*Stream which is on top of > File*Stream. The playback file is big enough, can be up to 3 GB a day > (hundreds of millions of events). > Everything works ok - all events encoded in the file can be replayed > back just fine. > Now I need to navigate through the playback (forth and back), kind of > rewinding. > I wanted to periodically insert special key frames that include such > information as byte offset of previous key frame (to rewind back) and > similar offset of the next key frame (to rewind forth), plus some > other key data. So I would navigate through the file faster by > skipping big portions of events between such key frames. > In order to maintain key frames linked each to other I need to move > back in the output buffer, modify previous key frame with correct > value of offset field and move forward to the initial position in > order to serialize next key frame and further events and so on. If > needed key frame can be of fixed size (e.g. sfixed64 types can be > used). > Reading and partial parsing of all events until good one is found is > not an option. > > How is this possible with protobuf implementation? e.g. > CodedOutputStream::Skip(int count) doesn't accept negative count as a > parameter. > > Perhaps there is alternate way to reach the goal - fast navigation in > the binary gzip'ed file that includes messages of variable length. > Any thoughts? > > Thank you in advance. > > /Roman > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
