Re: [s3ql] Recovery using local cache and s3ql_data_ only

[Nikolaus Rath]

On Oct 12 2016, Nikhil Choudhary  wrote:
>> The s3ql_passphrase object can be recovered from the master key, but
>> currently there is no code to do that.
>
> Not so good to hear, if this is the roadblock that it sounds like, I’m
> in for a restoration from scratch.

Try the attached patch, it adds a 'recover' option to s3qladm.

$ s3qladm recover local://bucket
Enter master key: 
KMWg kgYd S5ni K9VQ fgzO tZcB nWvI KA1I q/1d P/ii bMY=
Enter new encryption password: 
Confirm new encryption password: 

Make sure to try it on a test file system first, obviously!


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
diff --git a/src/s3ql/adm.py b/src/s3ql/adm.py
--- a/src/s3ql/adm.py
+++ b/src/s3ql/adm.py
@@ -19,7 +19,9 @@
 from datetime import datetime as Datetime
 from getpass import getpass
 from contextlib import contextmanager
+from base64 import b64decode
 import os
+import re
 import shutil
 import functools
 import sys
@@ -48,6 +50,8 @@
   parents=[pparser])
 subparsers.add_parser("clear", help="delete file system and all data",
   parents=[pparser])
+subparsers.add_parser("recover", help="Recover master key",
+  parents=[pparser])
 subparsers.add_parser("download-metadata",
   help="Interactively download metadata backups. "
"Use only if you know what you are doing.",
@@ -87,6 +91,10 @@
 with get_backend(options, raw=True) as backend:
 return clear(backend, options)
 
+if options.action == 'recover':
+with get_backend(options, raw=True) as backend:
+return recover(backend, options)
+
 with get_backend(options) as backend:
 if options.action == 'upgrade':
 return upgrade(backend, get_backend_cachedir(options.storage_url,
@@ -167,6 +175,27 @@
 backend['s3ql_passphrase_bak3'] = data_pw
 backend.passphrase = data_pw
 
+def recover(backend, options):
+print("Enter master key: ")
+data_pw = sys.stdin.readline()
+data_pw = re.sub(r'\s+', '', data_pw)
+data_pw = b64decode(data_pw)
+assert len(data_pw)== 32
+
+if sys.stdin.isatty():
+wrap_pw = getpass("Enter new encryption password: ")
+if not wrap_pw == getpass("Confirm new encryption password: "):
+raise QuietError("Passwords don't match")
+else:
+wrap_pw = sys.stdin.readline().rstrip()
+wrap_pw = wrap_pw.encode('utf-8')
+
+backend = ComprencBackend(wrap_pw, ('lzma', 2), backend)
+backend['s3ql_passphrase'] = data_pw
+backend['s3ql_passphrase_bak1'] = data_pw
+backend['s3ql_passphrase_bak2'] = data_pw
+backend['s3ql_passphrase_bak3'] = data_pw
+
 def clear(backend, options):
 print('I am about to delete all data in %s.' % backend,
   'This includes any S3QL file systems as well as any other stored objects.',

Re: [s3ql] Recovery using local cache and s3ql_data_ only

[Nikolaus Rath]

On Oct 12 2016, Nikhil Choudhary  wrote:
>> On Oct 12, 2016, at 3:41 PM, Nikolaus Rath  wrote:
>> 
>> So the only objects that are lost are s3ql_metadata, s3ql_passphrase
>> s3ql_seq_no no?
>
> Hello Nikolaus, thanks for the quick response!  This is correct.
>
>> As long as at least one s3ql_seq_no object is present, this should not
>> pose a problem.
>
> Does the particular object matter or is any one sufficient?  I
> recovered s3ql_seq_no_[4,5,6,8,10,13]

Any one will do.

>> If the s3ql_metadata object is missing, fsck.s3ql can recreate it from
>> the .db file. You may have to comment out one or two lines in fsck.s3ql
>> to deal with the complete absence of existing metadata.
>
> Good to hear, I looked into the patches that you’ve posted previously for 
> prioritizing local cache.
>
>> The s3ql_passphrase object can be recovered from the master key, but
>> currently there is no code to do that.
>
> Not so good to hear, if this is the roadblock that it sounds like, I’m
> in for a restoration from scratch.

I'll see if I can find a few minutes to send you a patch. It shouldn't
be too complicated.

>> Sounds like a terrible idea. What are you trying to achieve with this?
>
> I’m not a fan of convoluted setups either, they seem prone to
> breakage.  That said, this is a way to get s3ql’s dedupe, encryption,
> and caching capabilities working with ACD.

Yeah, but why don't you simply use S3 instead of ACD? 


Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [s3ql] Recovery using local cache and s3ql_data_ only

[Nikhil Choudhary]

Hello Nikolaus, thanks for the quick response! 

> So the only objects that are lost are s3ql_metadata, s3ql_passphrase
> s3ql_seq_no no?

Correct.

> As long as at least one s3ql_seq_no object is present, this should not
> pose a problem.

Does the particular object matter or is any one sufficient?  I recovered 
s3ql_seq_no_[4,5,6,8,10,13]

> If the s3ql_metadata object is missing, fsck.s3ql can recreate it from
> the .db file. You may have to comment out one or two lines in fsck.s3ql
> to deal with the complete absence of existing metadata.

Good to hear, I looked into the patches that you’ve posted previously for 
prioritizing local cache.

> The s3ql_passphrase object can be recovered from the master key, but
> currently there is no code to do that.

Not so good to hear, if this is the roadblock that it sounds like, I’m in for a 
restoration from scratch.

> Sounds like a terrible idea. What are you trying to achieve with this?

I’m not a fan of convoluted setups either, they seem prone to breakage.  That 
said, this is a way to get s3ql’s dedupe, encryption, and caching capabilities 
working with ACD.  Surprisingly this test configuration’s performance is more 
than sufficient for my use: personal video that doesn’t change and will only be 
infrequently read at 5-6mbit, and perhaps once a week have new files added.  
The user-facing functionality gap is that the filesystem is offline while the 
local files are synced to acd, otherwise I haven’t run across any issues (other 
than user-stupidity).  If this continues to work, I’m interested in learning to 
code a bit more and take a stab at adding a real ACD backend to s3ql.

Thanks for s3ql, it really does look great.
-Nikhil

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [s3ql] Recovery using local cache and s3ql_data_ only

[Nikhil Choudhary]

Hello Nikolaus, thanks for the quick response! Apologies if this is a 
duplicate, replying directly to s3ql@googlegroups only didn’t show up in the 
web UI.

> So the only objects that are lost are s3ql_metadata, s3ql_passphrase
> s3ql_seq_no no?

Correct.

> As long as at least one s3ql_seq_no object is present, this should not
> pose a problem.

Does the particular object matter or is any one sufficient?  I recovered 
s3ql_seq_no_[4,5,6,8,10,13]

> If the s3ql_metadata object is missing, fsck.s3ql can recreate it from
> the .db file. You may have to comment out one or two lines in fsck.s3ql
> to deal with the complete absence of existing metadata.

Good to hear, I looked into the patches that you’ve posted previously for 
prioritizing local cache.

> The s3ql_passphrase object can be recovered from the master key, but
> currently there is no code to do that.

Not so good to hear, if this is the roadblock that it sounds like, I’m in for a 
restoration from scratch.

> Sounds like a terrible idea. What are you trying to achieve with this?

I’m not a fan of convoluted setups either, they seem prone to breakage.  That 
said, this is a way to get s3ql’s dedupe, encryption, and caching capabilities 
working with ACD.  Surprisingly this test configuration’s performance is more 
than sufficient for my use: personal video that doesn’t change and will only be 
infrequently read at 5-6mbit, and perhaps once a week have new files added.  
The user-facing functionality gap is that the filesystem is offline while the 
local files are synced to acd, otherwise I haven’t run across any issues (other 
than user-stupidity).  If this continues to work, I’m interested in learning to 
code a bit more and take a stab at adding a real ACD backend to s3ql.

Thanks for s3ql, it really does look great.
-Nikhil

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [s3ql] Recovery using local cache and s3ql_data_ only

[Nikolaus Rath]

On Oct 12 2016, Nikhil Choudhary  wrote:
> Hi folks,
>
> Is it possible to recover a filesystem with an up-to-date local cache
> (.db and .params files), s3ql_data_ files, and the encryption master
> key?

That depends on what you want to recover from.

> I’m using s3ql with local storage backend (details below) - after
> cleanly unmounting the filesystem, I made a critical error and
> incorrectly deleted the s3ql files in the backend directory, except
> the s3ql_data_ directory.  Unfortunately, extundelete and other
> utilities were only able to grab some of the older s3ql_metadata_bak
> and s3ql_seq_no files, not the most recent s3ql_metadata file or the
> passphrase file.

So the only objects that are lost are s3ql_metadata, s3ql_passphrase
s3ql_seq_no no?

As long as at least one s3ql_seq_no object is present, this should not
pose a problem.

If the s3ql_metadata object is missing, fsck.s3ql can recreate it from
the .db file. You may have to comment out one or two lines in fsck.s3ql
to deal with the complete absence of existing metadata.

The s3ql_passphrase object can be recovered from the master key, but
currently there is no code to do that.

> Background: I’m testing s3ql along with overlay and acd_cli’s FUSE
> mount for Amazon Cloud Drive.  s3ql’s local storage backend points to
> an overlay mountpoint with acd_cli as a read-only layer and a local
> directory as the rw layer.  Files are copied to the s3ql mountpoint,
> and periodically I unmount s3ql and acd, sync the local s3ql_data_
> directory to acd s3ql_data_, delete the local contents of s3ql_data_,
> and re-mount acd and s3ql.

Sounds like a terrible idea. What are you trying to achieve with this?

Best,
-Nikolaus
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[s3ql] Recovery using local cache and s3ql_data_ only

[Nikhil Choudhary]

Hi folks,

Is it possible to recover a filesystem with an up-to-date local cache (.db and 
.params files), s3ql_data_ files, and the encryption master key?

I’m using s3ql with local storage backend (details below) - after cleanly 
unmounting the filesystem, I made a critical error and incorrectly deleted the 
s3ql files in the backend directory, except the s3ql_data_ directory.  
Unfortunately, extundelete and other utilities were only able to grab some of 
the older s3ql_metadata_bak and s3ql_seq_no files, not the most recent 
s3ql_metadata file or the passphrase file.

The last unmount action:
Oct 12 09:47:07 mount.s3ql[14294]: mount.s3ql[14294:Metadata-Upload-Thread] 
s3ql.mount.run: File system unchanged, not uploading metadata.
Oct 12 09:47:08 mount.s3ql[14294]: mount.s3ql[14294:MainThread] 
s3ql.mount.main: FUSE main loop terminated.
Oct 12 09:47:08 mount.s3ql[14294]: mount.s3ql[14294:MainThread] 
s3ql.mount.unmount: Unmounting file system...
Oct 12 09:47:08 mount.s3ql[14294]: mount.s3ql[14294:MainThread] 
s3ql.mount.main: File system unchanged, not uploading metadata.
Oct 12 09:47:08 mount.s3ql[14294]: mount.s3ql[14294:MainThread] 
s3ql.mount.main: Cleaning up local metadata...
Oct 12 09:47:08 mount.s3ql[14294]: mount.s3ql[14294:MainThread] 
s3ql.mount.main: All done.

Cache dir:
20717568 Oct 12 09:47 local:=2F=2F=2Fs3ql=2Fs3ql-overlay=2F.db
197 Oct 12 09:47 local:=2F=2F=2Fs3ql=2Fs3ql-overlay=2F.params
0 Oct  9 15:42 mount.s3ql_crit.log

Recovered files:
2628246 Oct 12 10:45 s3ql_metadata_bak_2
257749 Oct 12 10:45 s3ql_metadata_bak_4
650 Oct 12 10:44 s3ql_metadata_bak_8
276 Oct 12 10:47 s3ql_seq_no_10
276 Oct 12 10:47 s3ql_seq_no_13
275 Oct 12 10:49 s3ql_seq_no_4
338 Oct 12 10:48 s3ql_seq_no_5
275 Oct 12 10:48 s3ql_seq_no_6
275 Oct 12 10:48 s3ql_seq_no_8

It would save a few weeks of work to be able to restore access to the 
filesystem - any and all help is appreciated!

Thanks,
Nikhil

Background: I’m testing s3ql along with overlay and acd_cli’s FUSE mount for 
Amazon Cloud Drive.  s3ql’s local storage backend points to an overlay 
mountpoint with acd_cli as a read-only layer and a local directory as the rw 
layer.  Files are copied to the s3ql mountpoint, and periodically I unmount 
s3ql and acd, sync the local s3ql_data_ directory to acd s3ql_data_, delete the 
local contents of s3ql_data_, and re-mount acd and s3ql.  This setup is still 
in testing, so everything except file deletion is scripted, and naturally 
that’s the portion I erred on - instead of deleting just the local s3ql_data_ 
contents, I wiped out the entire contents of the local s3ql directory.  The 
s3ql_data_ files are completely up to date on acd and safe, but the metadata, 
passphrase, and other files in the root of the s3ql mount are gone.  

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.