On Jun 29 2019, Christian Loitsch <[email protected]> wrote: > Hi Nikolaus! > > On Thu, 27 Jun 2019 at 21:56, Nikolaus Rath <[email protected]> wrote: > [...] > >> On Jun 27 2019, Christian Loitsch <[email protected]> wrote: >> >> On Jun 26 2019, Daniel Jagszent <[email protected]> wrote: >> >> >> [...] During the update files were missing and I decided to copy >> >> >> everything to a local disk before continuing the upgrade process. >> >> >> [...] >> >> > did you copy the files from an object storage (S3, Google Storage, >> >> > OpenStack Swift)? That does not work. S3QL needs metadata of the >> >> > copied "files" that you probably did not copy over – and even if, the >> >> > Local file backend stores the metadata differently, AFAIK. >> >> >> >> That's right. >> >> >> >> > You cannot easily copy a S3QL filesystem from one backend type to >> >> > another. >> > I am currently trying to retrieve the metadata. > AFAIKT its: > X-Object-Meta-000: 'format_version': 2, > X-Object-Meta-001: 'compression': 'ZLIB', > X-Object-Meta-002: 'encryption': 'None', > X-Object-Meta-003: 'data': b'eyAgfQ==', > X-Object-Meta-004: 'needs_reupload': True, > X-Object-Meta-Format: raw2 > X-Object-Meta-Md5: q0cp30i91gdvcDkh5w2rsw== > > for all files. (Even Meta-Md5 is identical) > > I would now try to rewrite the 'local' Backend to return those values: > Rewrite _read_meta(fh) to not read anything from the file but retrieve the > metadata from somewhere else. > > But before investing to much work: is this even plausible?
Yeah, in principle that's possible. > I think I've never done anything other than fsck / mount on the "online" > filesystem, but especially the > X-Object-Meta-002: 'encryption': 'None', > looks incorrect. Well, it looks like you haven't enabled encryption. > > It should use encryption: > === ~/.s3ql/authinfo2 === > [s3ql] > [...] > fs-passphrase: B5mJhEEQu2MhGR8yBh55L > ====== mount.s3ql doesn't try to use the passphrase if the filesystem isn't encrypted. >> >> It's pretty easy if you use the right tools :-). >> >> >> >> https://github.com/s3ql/s3ql/blob/master/contrib/clone_fs.py > > I would really like to use clone_fs.py the connection is simply not stable > enough. Not sure I follow. It should retry automatically on network issues. If you were able to download everything with other tools, then you should be able to do it with clone_fs.py too. Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/8736jrtow2.fsf%40vostro.rath.org. For more options, visit https://groups.google.com/d/optout.
