[s3ql] Recovery using local cache and s3ql_data_ only
Hi folks, Is it possible to recover a filesystem with an up-to-date local cache (.db and .params files), s3ql_data_ files, and the encryption master key? I’m using s3ql with local storage backend (details below) - after cleanly unmounting the filesystem, I made a critical error and incorrectly deleted the s3ql files in the backend directory, except the s3ql_data_ directory. Unfortunately, extundelete and other utilities were only able to grab some of the older s3ql_metadata_bak and s3ql_seq_no files, not the most recent s3ql_metadata file or the passphrase file. The last unmount action: Oct 12 09:47:07 mount.s3ql[14294]: mount.s3ql[14294:Metadata-Upload-Thread] s3ql.mount.run: File system unchanged, not uploading metadata. Oct 12 09:47:08 mount.s3ql[14294]: mount.s3ql[14294:MainThread] s3ql.mount.main: FUSE main loop terminated. Oct 12 09:47:08 mount.s3ql[14294]: mount.s3ql[14294:MainThread] s3ql.mount.unmount: Unmounting file system... Oct 12 09:47:08 mount.s3ql[14294]: mount.s3ql[14294:MainThread] s3ql.mount.main: File system unchanged, not uploading metadata. Oct 12 09:47:08 mount.s3ql[14294]: mount.s3ql[14294:MainThread] s3ql.mount.main: Cleaning up local metadata... Oct 12 09:47:08 mount.s3ql[14294]: mount.s3ql[14294:MainThread] s3ql.mount.main: All done. Cache dir: 20717568 Oct 12 09:47 local:=2F=2F=2Fs3ql=2Fs3ql-overlay=2F.db 197 Oct 12 09:47 local:=2F=2F=2Fs3ql=2Fs3ql-overlay=2F.params 0 Oct 9 15:42 mount.s3ql_crit.log Recovered files: 2628246 Oct 12 10:45 s3ql_metadata_bak_2 257749 Oct 12 10:45 s3ql_metadata_bak_4 650 Oct 12 10:44 s3ql_metadata_bak_8 276 Oct 12 10:47 s3ql_seq_no_10 276 Oct 12 10:47 s3ql_seq_no_13 275 Oct 12 10:49 s3ql_seq_no_4 338 Oct 12 10:48 s3ql_seq_no_5 275 Oct 12 10:48 s3ql_seq_no_6 275 Oct 12 10:48 s3ql_seq_no_8 It would save a few weeks of work to be able to restore access to the filesystem - any and all help is appreciated! Thanks, Nikhil Background: I’m testing s3ql along with overlay and acd_cli’s FUSE mount for Amazon Cloud Drive. s3ql’s local storage backend points to an overlay mountpoint with acd_cli as a read-only layer and a local directory as the rw layer. Files are copied to the s3ql mountpoint, and periodically I unmount s3ql and acd, sync the local s3ql_data_ directory to acd s3ql_data_, delete the local contents of s3ql_data_, and re-mount acd and s3ql. This setup is still in testing, so everything except file deletion is scripted, and naturally that’s the portion I erred on - instead of deleting just the local s3ql_data_ contents, I wiped out the entire contents of the local s3ql directory. The s3ql_data_ files are completely up to date on acd and safe, but the metadata, passphrase, and other files in the root of the s3ql mount are gone. -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [s3ql] Recovery using local cache and s3ql_data_ only
Hello Nikolaus, thanks for the quick response! Apologies if this is a duplicate, replying directly to s3ql@googlegroups only didn’t show up in the web UI. > So the only objects that are lost are s3ql_metadata, s3ql_passphrase > s3ql_seq_no no? Correct. > As long as at least one s3ql_seq_no object is present, this should not > pose a problem. Does the particular object matter or is any one sufficient? I recovered s3ql_seq_no_[4,5,6,8,10,13] > If the s3ql_metadata object is missing, fsck.s3ql can recreate it from > the .db file. You may have to comment out one or two lines in fsck.s3ql > to deal with the complete absence of existing metadata. Good to hear, I looked into the patches that you’ve posted previously for prioritizing local cache. > The s3ql_passphrase object can be recovered from the master key, but > currently there is no code to do that. Not so good to hear, if this is the roadblock that it sounds like, I’m in for a restoration from scratch. > Sounds like a terrible idea. What are you trying to achieve with this? I’m not a fan of convoluted setups either, they seem prone to breakage. That said, this is a way to get s3ql’s dedupe, encryption, and caching capabilities working with ACD. Surprisingly this test configuration’s performance is more than sufficient for my use: personal video that doesn’t change and will only be infrequently read at 5-6mbit, and perhaps once a week have new files added. The user-facing functionality gap is that the filesystem is offline while the local files are synced to acd, otherwise I haven’t run across any issues (other than user-stupidity). If this continues to work, I’m interested in learning to code a bit more and take a stab at adding a real ACD backend to s3ql. Thanks for s3ql, it really does look great. -Nikhil -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [s3ql] Recovery using local cache and s3ql_data_ only
Hello Nikolaus, thanks for the quick response! > So the only objects that are lost are s3ql_metadata, s3ql_passphrase > s3ql_seq_no no? Correct. > As long as at least one s3ql_seq_no object is present, this should not > pose a problem. Does the particular object matter or is any one sufficient? I recovered s3ql_seq_no_[4,5,6,8,10,13] > If the s3ql_metadata object is missing, fsck.s3ql can recreate it from > the .db file. You may have to comment out one or two lines in fsck.s3ql > to deal with the complete absence of existing metadata. Good to hear, I looked into the patches that you’ve posted previously for prioritizing local cache. > The s3ql_passphrase object can be recovered from the master key, but > currently there is no code to do that. Not so good to hear, if this is the roadblock that it sounds like, I’m in for a restoration from scratch. > Sounds like a terrible idea. What are you trying to achieve with this? I’m not a fan of convoluted setups either, they seem prone to breakage. That said, this is a way to get s3ql’s dedupe, encryption, and caching capabilities working with ACD. Surprisingly this test configuration’s performance is more than sufficient for my use: personal video that doesn’t change and will only be infrequently read at 5-6mbit, and perhaps once a week have new files added. The user-facing functionality gap is that the filesystem is offline while the local files are synced to acd, otherwise I haven’t run across any issues (other than user-stupidity). If this continues to work, I’m interested in learning to code a bit more and take a stab at adding a real ACD backend to s3ql. Thanks for s3ql, it really does look great. -Nikhil -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
