On 2020-07-08 at 15:39 -0700, David Gasaway wrote: > On Wed, Jul 8, 2020 at 6:46 AM Ivan Shapovalov <[email protected]> > wrote: > > > > This specific filesystem is used to store a borg repository. > > > > I currently use borg + s3ql myself. I don't think I'd recommend > it. In my > experience, borg check operations don't fare well in this scenario > (as in, > downloads of the entire repository). I plan to switch to > s3ql_backup.py + > s3ql, but the prospect of the uploads needed to start from scratch > are > daunting. >
If I understand you right, then a `borg check` _has_ to download the entire repository, because, well, it's an integrity check :) But yeah, I don't hold any illusions — though I don't see much of a choice either. I have a 8+ TB dataset containing some VMs, some random assorted small files which come and go, some user shares with arbitrary data and I need to snapshot+backup all that stuff for point-in-time recovery. Using S3QL's built-in quasi-snapshotting is a possibility, but I'm not sure if rsync is going to be to any efficient with delta transfers of VM images. I guess I'll try my luck with various block sizes, see how large the metadata grows and how efficient borg reads will be. I still need to profile borg I/O patterns rigorously, but I believe that small random reads are the main issue. Worst case I guess I'll have to write a custom caching layer for partial blocks, maybe borg-aware. -- Ivan Shapovalov / intelfx / -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/709919d29857d843aea829e35939f45fae7071ef.camel%40intelfx.name.
