So, a couple months ago, I made a mistake and deleted some data... I'm going to share the experience here, and outline some of the plans I've got to move on. Help would be welcome, but mostly my aim here is to provide some insight as to how things go when they go wrong, for considerations on improvements.
I was working on testing out configuring Digital Ocean's new Spaces product as various backing store to Perkeep (well, a version that was still camlistore). At some point during the process, I must have deleted some blobpacked files on my server. I know I was making odd rsync commands at one point that day, but I don't know what did it. Whatever it was it deleted an early segment of blobs starting with "sha1-000" up through about "sha1-004". At the time, I didn't realize. Perkeep must have had blobs cached, and so things were fine in the UI. I had eventually set aside work on that Spaces implementation, because of (DO's) performance problems. Then, the other day, I went to go update to a more recent version (to Perkeep!). After a little config updating, I eventually got it running, but the UI wasn't showing my content properly. It just looked like a huge list of folders, with no names and no meaningful contents (just poking at a few). Searches I used to run in the UI don't return any results. I ran reindexing and eventually did both recovery modes. That's when I started seeing the errors about missing blobs, all in that early range of sha1s. May 6 10:51:22 new perkeepd[2237]: 2018/05/06 10:51:22 Error reindexing > sha1-0009743ea7ead6126510ad334cfe4199c2c383f7: index: failed to fetch > sha1-0009743ea7ead6126510ad334cfe4199c2c383f7 for reindexing: file does not > exist So my first observation here: problems on the backend storage can easily go unseen. Earlier detection of the problem may have allowed me to recover from cache. The second observation: recovery/reindex errors cause the instance to fail to start with limited info for repair: (what's telling it to reindex something that doesn't exist?). So the situation I find myself in now: I deleted about 0.2% of my data, but the instance is more or less hosed. (And the data is just gone: this is how I realized my backups aren't covering the block storage I moved my perkeep data off to... oops). I have a few years of data in that instance. I have a few importers: google photos and rss feeds. I have a few attempts at syncing my music and whole home directories up to it. Then there's just some odds and ends uploaded via the UI. Those are the important bits that would be great to recover, mostly because I'm not sure what might be there, and the rest is recoverable elsewhere. The crossroads is between: clean up the surviving data to repair the instance or search for non-re-importable data and starting with a fresh new instance. I'm not sure what the process for repairing the instance would be... when the index complains about blobs being missing, what causes it to expect those blobs? Is it possible to look up blobs which reference other specific (but non-existent) blobs? I'm not sure what schema relations to interrogate, so searching for non-re-importable data sounds easier. I can start to build up a search query full of exclusions until I've gotten rid of all the imported data. That's what I intend to try first. That said, my third observation: there doesn't seem to be a good way to analyze a perkeep instance's data in aggregate without a lot of manual labor. (Or I just haven't seen it yet?) I'm not sure what, if any, good improvements we could make to perkeep based on this information, but I'm happy to keep discussing or share more as my discovery/recovery process continues. -- You received this message because you are subscribed to the Google Groups "Perkeep" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
