I'm impressed using s3ql (v 3.8.1) - it does (nearly) exactly what I want!
I've run into two issues recently with s3 metadata - one I've managed to fix, the other I'm looking for advice.
When syncing an s3 bucket containing an s3ql file system to a new one, and then trying to mount the new one, I was getting errors with the metadata. The reason was that aws sync will lose metadata if it uses multipart transfers.
I fixed this using "aws configure set default.s3.multipart_threshold 20MB" to increase the multipart threshold above the size of the s3ql data blocks.
I also used "aws s3 sync --metadata-directive COPY" but I believe that is the default anyway, so it was probably superfluous.
Then I tried to use AWS Backup on s3 buckets to do a similar job. This is where the problem came in - the ETag is not always preserved on restore, and is not an MD5 when multipart uploads get used. The decision on whether AWS uses multi-part uploads on the restore seems to be out of my control.
I've not yet looked into the internals of s3ql - is it possible to fix the ETag issue? I know ETags can't be changed, but could s3ql recover from the ETags changing?
Thanks, Pete. -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/8cf45e3e-d5c0-584a-467d-ab2cdd18d0bf%40goteck.co.uk.
