a separate file only makes things worse as you have to deal with the possibility of corruption in either one of the files or both...
I think a better approach is to start with the TFile format, prefix each chunk in the file (sorry, it;s been some time since my last read about this interesting format) with its length, then write the mtd+index at the end when you close the file and add some kind of magic number /+ signature at the end. when you open the file, check the magic/signature/both, if it's valid you can trust the index to 'random access' blocks in the file, if it's not you can take the sequential path, based on the lengths that prefix each block in the file. Eyal. On Friday, May 18, 2012 7:05:38 PM UTC+3, Igor Gatis wrote: > > Is a separate index file an option? You could build main file and this > index file. If one file only is desirable, you can prepend or append index > later. > > On Thu, May 17, 2012 at 8:22 AM, Roman Vinogradov > <.<[email protected]> > ..> wrote: > >> Thank you all. >> I think TFile is very similar to what I am trying to do. >> One issue of TFile is that it stores all meta and indices in its tail >> which in turn means that if the writing process suddenly crashes for >> some reason (e.g. hardware issue or something not related to our >> process) then file will be of incomplete format and it won't be >> possible to read it or even recover it. >> >> /Roman >> >> On 17 май, 10:28, Eyal Farago <...> wrote: >> > you can use an approach like HADDOP's TFilehttps:// >> issues.apache.org/jira/secure/attachment/12396286/TFile%20Spe... >> > >> > basically they store compressed chunks in a file, and at the end of the >> > file (hence T(ail)File) they store come kind of a sparse index to >> > the chunks. >> > >> > Eyal. >> > >> > >> > >> > - Скрыть цитируемый текст - >> > >> > - Показать цитируемый текст - >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/protobuf?hl=en. >> >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To view this discussion on the web visit https://groups.google.com/d/msg/protobuf/-/82mO_OpP1UkJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
