Rich Freeman wrote: > On Sun, Aug 14, 2022 at 6:44 PM Dale <[email protected]> wrote: >> Right now, I'm using rsync which doesn't compress files but does just >> update things that have changed. I'd like to find some way, software >> but maybe there is already a tool I'm unaware of, to compress data and >> work a lot like rsync otherwise. > So, how important is it that it work exactly like rsync? > > I use duplicity, in part because I've been using it forever. Restic > seems to be a similar program most are using these days which I > haven't looked at super-closely but I'd look at that first if starting > out. > > Duplicity uses librsync, so it backs up exactly the same data as rsync > would, except instead of replicating entire files, it creates streams > of data more like something like tar. So if you back up a million > small files you might get out 1-3 big files. It can compress and > encrypt the data as you wish. The downside is that you don't end up > with something that looks like your original files - you have to run > the restore process to extract them all back out. It is extremely > space-efficient though - if 1 byte changes in the middle of a 10GB > file you'll end up just backing up maybe a kilobyte or so (whatever > the block size is), which is just like rsync. > > Typically you rely on metadata to find files that change which is > fast, but I'm guessing you can tell these programs to do a deep scan > which of course requires reading the entire contents, and that will > discover anything that was modified without changing ctime/mtime. > > The output files can be split to any size, and the index info (the > metadata) is separate from the raw data. If you're storing to > offline/remote/cloud/whatever storage typically you keep the metadata > cached locally to speed retrieval and to figure out what files have > changed for incrementals. However, if the local cache isn't there > then it will fetch just the indexes from wherever it is stored > (they're small). > > It has support for many cloud services - I store mine to AWS S3. > > There are also some options that are a little closer to rsync like > rsnapshot and burp. Those don't store compressed (unless there is an > option for that or something), but they do let you rotate through > multiple backups and they'll set up hard links/etc so that they are > de-duplicated. Of course hard links are at the file level so if 1 > byte inside a file changes you'll end up with two full copies. It > will still only transfer a single block so the bandwidth requirements > are similar to rsync. >
Duplicity sounds interesting except that I already have the drive encrypted. Keep in mind, these are external drives that I hook up long enough to complete the backups then back in a fire safe they go. The reason I mentioned being like rsync, I don't want to rebuild a backup from scratch each time as that would be time consuming. I thought of using Kbackup ages ago and it rebuilds from scratch each time but it does have the option of compressing. That might work for small stuff but not many TBs of it. Back in the early 90's, I remember using a backup software that was incremental. It would only update files that changed and would do it over several floppy disks and compressed it as well. Something like that nowadays is likely rare if it exists at all since floppies are long dead. I either need to split my backup into two pieces or compress my data. That is why I mentioned if there is a way to backup first part of alphabet in one command, switch disks and then do second part of alphabet to another disk. Mostly, I just want to add compression to what I do now. I figure there is a tool for it but no idea what it is called. Another method is splitting into two parts. In the long run, either should work and may end up needing both at some point. :/ If I could add both now, save me some problems later on. I guess. I might add, I also thought about using a Raspberry Pi thingy and having sort of a small scale NAS thing. I'm not sure about that thing either tho. Plus, they pricey right now. $$$ Dale :-) :-)
