Similarly to what Jason suggested, you may have summary blocks which
lets you quick inspect value ranges. Like:
message Record {
optional int id = 1;
// fields only you know about.
}
message RecordBlock {
repeated Record record = 1; // You could declare these as 'bytes' type
// instead of 'Record'
type. This avoids parsing
// each Record if you're
not going to use it.
optional int max_id = 2;
optional int min_id = 3;
}
message RecordsFile {
repeated RecordBlock raw_blocks = 1; // Readly available RecordBlock.
repeated bytes gziped_blocks = 2; // Gzip encoded RecordBlock.
repeated bytes other_encoding_blocks = 3; // Other encoded RecordBlock.
}
You may use RecordsFile to append records to your file. For parsing,
you can use CodedInputStream.
On Tue, Apr 3, 2012 at 15:26, Jason Hsueh <[email protected]> wrote:
>
> In general protobuf doesn't try to provide these kinds of container formats.
> Gzip*Stream is meant to deal with data that is valid gzip; inserting frames
> to make it searchable would break the format. In the archives you may find
> other references to the internal container file format that Google has; this
> effectively has blocks of individual records, and it's possible to skip
> blocks or seek to a position (or the next valid record after a position). It
> sounds like you want something similar, with your events as the records, and
> each block as the 'key frames'. Rather than compressing the entire file, you
> could perform block level compression, and treat each individual record as a
> separate stream that can be decoded into your protobuf.
>
>
> On Mon, Apr 2, 2012 at 5:30 AM, Roman Vinogradov <[email protected]>
> wrote:
>>
>> Hello,
>>
>> I use protobuf to store events in playback file for further replaying.
>> I use Coded*Stream on top of Gzip*Stream which is on top of
>> File*Stream. The playback file is big enough, can be up to 3 GB a day
>> (hundreds of millions of events).
>> Everything works ok - all events encoded in the file can be replayed
>> back just fine.
>> Now I need to navigate through the playback (forth and back), kind of
>> rewinding.
>> I wanted to periodically insert special key frames that include such
>> information as byte offset of previous key frame (to rewind back) and
>> similar offset of the next key frame (to rewind forth), plus some
>> other key data. So I would navigate through the file faster by
>> skipping big portions of events between such key frames.
>> In order to maintain key frames linked each to other I need to move
>> back in the output buffer, modify previous key frame with correct
>> value of offset field and move forward to the initial position in
>> order to serialize next key frame and further events and so on. If
>> needed key frame can be of fixed size (e.g. sfixed64 types can be
>> used).
>> Reading and partial parsing of all events until good one is found is
>> not an option.
>>
>> How is this possible with protobuf implementation? e.g.
>> CodedOutputStream::Skip(int count) doesn't accept negative count as a
>> parameter.
>>
>> Perhaps there is alternate way to reach the goal - fast navigation in
>> the binary gzip'ed file that includes messages of variable length.
>> Any thoughts?
>>
>> Thank you in advance.
>>
>> /Roman
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected].
>> For more options, visit this group at
>> http://groups.google.com/group/protobuf?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
