[protobuf] Fast navigation in binary gzip'ed file that includes messages of variable length

2012-04-02 Thread Roman Vinogradov
Hello, I use protobuf to store events in playback file for further replaying. I use Coded*Stream on top of Gzip*Stream which is on top of File*Stream. The playback file is big enough, can be up to 3 GB a day (hundreds of millions of events). Everything works ok - all events encoded in the file can

[protobuf] Re: Fast navigation in binary gzip'ed file that includes messages of variable length

2012-05-17 Thread Roman Vinogradov
Thank you all. I think TFile is very similar to what I am trying to do. One issue of TFile is that it stores all meta and indices in its tail which in turn means that if the writing process suddenly crashes for some reason (e.g. hardware issue or something not related to our process) then file will

[protobuf] Re: Fast navigation in binary gzip'ed file that includes messages of variable length

2012-05-21 Thread Roman Vinogradov
Gatis wrote: > > > Is a separate index file an option? You could build main file and this > > index file. If one file only is desirable, you can prepend or append index > > later. > > > On Thu, May 17, 2012 at 8:22 AM, Roman Vinogradov > > <. > > ..>