Reading some more, I discovered the old Camlistore group and found this thread about integrity checking:
https://groups.google.com/forum/#!searchin/camlistore/graph|sort:date/camlistore/_KiSv7x1Eh0/w_lik623AwAJ So it's good to know that reindexing will verify blobs have not been corrupted. But if this is done infrequently it still seems possible to lose data over time due to corruption. Some sort of auto-healing mechanism seems like a good feature for storage that is intended to last 100 years. Is anybody aware of any work that's been done on this for Perkeep? Thanks! On Saturday, May 18, 2019 at 2:53:19 PM UTC+1, Dan Cutting wrote: > > Hi, does Perkeep offer any sort of error detection or correction? > > Data sitting on a disk will "rot" over time (bad disks, bad drivers, > cosmic rays), and it would be good to automatically correct it. > > I'm thinking something along the lines of a Reed-Solomon-style blob server. > > A simpler possibility might be to rely on the fact that a blob is > content-addressed and check that the data read hashes back to the content > address before returning it to the client. Combining this with multiple > replicas would let the blob server replace bad blobs with good ones when > detected. > > Or does Perkeep already have a story around this kind of thing? I haven't > been able to find it. > > Thanks! > Dan > -- You received this message because you are subscribed to the Google Groups "Perkeep" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/perkeep/0ab8636f-5e20-454f-86de-160184ba7b14%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
