Re: [gentoo-user] Suggestions for backup scheme?
On 8/2/24 06:36, Frank Steinmetzger wrote: Am Tue, Jan 30, 2024 at 06:15:09PM - schrieb Grant Edwards: I need to set up some sort of automated backup on a couple Gentoo machines (typical desktop software development and home use). One of them used rsnapshot in the past but the crontab entries that drove that have vanished :/ (presumably during a reinstall or upgrade -- IIRC, it took a fair bit of trial and error to get the crontab entries figured out). I believe rsnapshot ran nightly and kept daily snapshots for a week, weekly snapshots for a month, and monthly snapshots for a couple years. Are there other backup solutions that people would like to suggest I look at to replace rsnapshot? I was happy enough with rsnapshot (when it was running), but perhaps there's something else I should consider? In my early backup times I, too, used rsnapshot to back up my ~ and rsync for my big media files. But that only included my PC. My laptop was wholly un-backed-up. I only syncronised much of my home and my audio collection between the two with unison. At some point my external 3 TB drive became free and then I started using borg to finally do proper backups. Borg is very similar to restic, I actually used the two in parallel for a while to compare them, but stayed with borg. One pain point was that I couln’t switch off restic’s own password protection. Since all my backup disks are LUKSed anyway, I don’t need that. Since borg works block-based, it does deduplication without extra cost and it is suitable for big image files which don’t change much. I do full filesystem backups of /, ~ and my media partition of my main PC and my laptop. I have one repository for each of those three filesystems, and each repo receives the data from both machines, so they are deduped. Since both machines run Arch, their roots are binary identical. The same goes for my unison-synced homes. Borg has retention logic built-in. You can say I want to keep the latest archive of each of the last 6 days/weeks/months/years, and it even goes down to seconds. And of course you can combine those rules. The only thing is they don’t overlap, meaning if you want to keep the last 14 days and the last four weeks, those weekly retentions start after the last daily snapshots. In summary, advantages: + fast dedup, built-in compression (different algos and levels configurable) + big data files allow for quick mirroring of repositories. I simply rsync my primary backup disk to two other external HDDs. + Incremental backups are quite fast because borg uses a cache to detect changed files quickly. Disadvantages: - you need borg to mount the backups it - it is not as fast as native disk access, especially during restore and when getting a total file listing due to lots of random I/O on the HDD. As example, I currently have 63 snapshots in my data partition repository: # borg list data/ tp_2021-06-07 Mon, 2021-06-07 16:27:44 [5f9ebd9f24353c340691b2a71f5228985a41699d2e23473ae4e9e795669c8440] kern_2021-06-07 Mon, 2021-06-07 23:58:56 [19c76211a9c35432e6a66ac1892ee19a08368af28d2d621f509af3d45f203d43] [... 55 more lines ...] kern_2024-01-14 Sun, 2024-01-14 20:53:23 [499ce7629e64cffb7ec6ec9ffbf0c595e4ede3d93f131a9a4b424b165647f645] tp_2024-01-14 Sun, 2024-01-14 20:57:42 [ea2baef3e4bb49c5aec7cf8536f7b00b55fb27ecae3a80ef9f5a5686a1da30d5] kern_2024-01-21 Sun, 2024-01-21 23:42:46 [71aa2ce6cf4021712f949af068498bfda7797b5d1c5ddc0f0ce8862b89e48961] tp_2024-01-21 Sun, 2024-01-21 23:48:24 [45e35ed9206078667fa62d0e4a1ac213e77f52415f196101d14ee21e79fc393d] kern_2024-02-04 Sun, 2024-02-04 23:16:43 [e1b015117143fad6b89cea66329faa888cffc990644e157b1d25846220c62448] tp_2024-02-04 Sun, 2024-02-04 23:23:15 [e9b167ceec1ab9a80cbdb1acf4ff31cd3935fc23e81674cad1b8694d98547aeb] The last “tp” (Thinkpad) snapshot contains 1 TB, “kern” (my PC) 809 GB. And here you see how much space this actually takes on disk: # borg info data/ [ ... ] Original size Compressed sizeDeduplicated size All archives: 56.16 TB 54.69 TB 1.35 TB Obviously, compression doesn’t do much for media files. But it is very effective in the repository for the root partitions: # borg info arch-root/ [ ... ] Original size Compressed sizeDeduplicated size All archives: 1.38 TB 577.58 GB 79.41 GB I would also like to add my +1 to borgbackup ... I long ago lost the ability to use snapshots and full size backups due to the sheer amount of data involved. Currently I use borg to backup multiple hosts to individual backups on a dedicated machine (low power arm based, 6TB drive). I also backup from the top level of the directory all those repos are stored in to another arm system (2TB drive) again using borg. As each 1st level backup only adds/changes a few chunks for each iteration, the second level only
Re: [gentoo-user] Suggestions for backup scheme?
Am Tue, Jan 30, 2024 at 06:15:09PM - schrieb Grant Edwards: > I need to set up some sort of automated backup on a couple Gentoo > machines (typical desktop software development and home use). One of > them used rsnapshot in the past but the crontab entries that drove > that have vanished :/ (presumably during a reinstall or upgrade -- > IIRC, it took a fair bit of trial and error to get the crontab entries > figured out). > > I believe rsnapshot ran nightly and kept daily snapshots for a week, > weekly snapshots for a month, and monthly snapshots for a couple > years. > > Are there other backup solutions that people would like to suggest I > look at to replace rsnapshot? I was happy enough with rsnapshot (when > it was running), but perhaps there's something else I should consider? In my early backup times I, too, used rsnapshot to back up my ~ and rsync for my big media files. But that only included my PC. My laptop was wholly un-backed-up. I only syncronised much of my home and my audio collection between the two with unison. At some point my external 3 TB drive became free and then I started using borg to finally do proper backups. Borg is very similar to restic, I actually used the two in parallel for a while to compare them, but stayed with borg. One pain point was that I couln’t switch off restic’s own password protection. Since all my backup disks are LUKSed anyway, I don’t need that. Since borg works block-based, it does deduplication without extra cost and it is suitable for big image files which don’t change much. I do full filesystem backups of /, ~ and my media partition of my main PC and my laptop. I have one repository for each of those three filesystems, and each repo receives the data from both machines, so they are deduped. Since both machines run Arch, their roots are binary identical. The same goes for my unison-synced homes. Borg has retention logic built-in. You can say I want to keep the latest archive of each of the last 6 days/weeks/months/years, and it even goes down to seconds. And of course you can combine those rules. The only thing is they don’t overlap, meaning if you want to keep the last 14 days and the last four weeks, those weekly retentions start after the last daily snapshots. In summary, advantages: + fast dedup, built-in compression (different algos and levels configurable) + big data files allow for quick mirroring of repositories. I simply rsync my primary backup disk to two other external HDDs. + Incremental backups are quite fast because borg uses a cache to detect changed files quickly. Disadvantages: - you need borg to mount the backups it - it is not as fast as native disk access, especially during restore and when getting a total file listing due to lots of random I/O on the HDD. As example, I currently have 63 snapshots in my data partition repository: # borg list data/ tp_2021-06-07 Mon, 2021-06-07 16:27:44 [5f9ebd9f24353c340691b2a71f5228985a41699d2e23473ae4e9e795669c8440] kern_2021-06-07 Mon, 2021-06-07 23:58:56 [19c76211a9c35432e6a66ac1892ee19a08368af28d2d621f509af3d45f203d43] [... 55 more lines ...] kern_2024-01-14 Sun, 2024-01-14 20:53:23 [499ce7629e64cffb7ec6ec9ffbf0c595e4ede3d93f131a9a4b424b165647f645] tp_2024-01-14 Sun, 2024-01-14 20:57:42 [ea2baef3e4bb49c5aec7cf8536f7b00b55fb27ecae3a80ef9f5a5686a1da30d5] kern_2024-01-21 Sun, 2024-01-21 23:42:46 [71aa2ce6cf4021712f949af068498bfda7797b5d1c5ddc0f0ce8862b89e48961] tp_2024-01-21 Sun, 2024-01-21 23:48:24 [45e35ed9206078667fa62d0e4a1ac213e77f52415f196101d14ee21e79fc393d] kern_2024-02-04 Sun, 2024-02-04 23:16:43 [e1b015117143fad6b89cea66329faa888cffc990644e157b1d25846220c62448] tp_2024-02-04 Sun, 2024-02-04 23:23:15 [e9b167ceec1ab9a80cbdb1acf4ff31cd3935fc23e81674cad1b8694d98547aeb] The last “tp” (Thinkpad) snapshot contains 1 TB, “kern” (my PC) 809 GB. And here you see how much space this actually takes on disk: # borg info data/ [ ... ] Original size Compressed sizeDeduplicated size All archives: 56.16 TB 54.69 TB 1.35 TB Obviously, compression doesn’t do much for media files. But it is very effective in the repository for the root partitions: # borg info arch-root/ [ ... ] Original size Compressed sizeDeduplicated size All archives: 1.38 TB 577.58 GB 79.41 GB -- Grüße | Greetings | Salut | Qapla’ Please do not share anything from, with or about me on any social network. “She understands. She doesn’t comprehend.” – River Tam, Firefly signature.asc Description: PGP signature
Re: [gentoo-user] Suggestions for backup scheme?
Le 30/01/2024 à 19:15, Grant Edwards a écrit : I need to set up some sort of automated backup on a couple Gentoo machines (typical desktop software development and home use). One of them used rsnapshot in the past but the crontab entries that drove that have vanished :/ (presumably during a reinstall or upgrade -- IIRC, it took a fair bit of trial and error to get the crontab entries figured out). I believe rsnapshot ran nightly and kept daily snapshots for a week, weekly snapshots for a month, and monthly snapshots for a couple years. Are there other backup solutions that people would like to suggest I look at to replace rsnapshot? I was happy enough with rsnapshot (when it was running), but perhaps there's something else I should consider? -- Grant I use restic [1] with the S3 backend. It manages snapshots and supports several backends. It is way faster than the previous backup solution I've used (dejadup). It came handy several times when I had to restore a specific file from a point in time. 1: https://packages.gentoo.org/packages/app-backup/restic
Re: [gentoo-user] Suggestions for backup scheme?
On Tue, Jan 30, 2024 at 3:08 PM Wol wrote: > > On 30/01/2024 19:19, Rich Freeman wrote: > > I'd echo the other advice. It really depends on your goals. > > If you just want a simple backup, I'd use something like rsync onto lvm > or btrfs or something. I've got a little script that sticks today's date > onto the snapshot name So, you've basically described what rsnapshot does, minus half the features. You should consider looking at it. It is basically an rsync wrapper and will automatically rotate multiple snapshots, and when it makes them they're all hard-linked such that they're as close to copy-on-write copies as possible. The result is that all those snapshots don't take up much space, unless your files are constantly changing. -- Rich
Re: [gentoo-user] Suggestions for backup scheme?
On 30/01/2024 19:19, Rich Freeman wrote: I'd echo the other advice. It really depends on your goals. If you just want a simple backup, I'd use something like rsync onto lvm or btrfs or something. I've got a little script that sticks today's date onto the snapshot name (used to snapshot / before I emerge :-) so if you run something like after each backup you know your snapshot is as it was that day. Rsync has a "backup in place option", so it will on;ly overwrite parts of the file that have changed, so if you use lvm or btrfs to snapshot the filesystem as part of the backup, you can then just mount that snapshot to get a complete filesystem image. Full backups for the price of incremental. Cheers, Wol
Re: [gentoo-user] Suggestions for backup scheme?
On Tue, Jan 30, 2024 at 1:15 PM Grant Edwards wrote: > > Are there other backup solutions that people would like to suggest I > look at to replace rsnapshot? I was happy enough with rsnapshot (when > it was running), but perhaps there's something else I should consider? > I'd echo the other advice. It really depends on your goals. I think the key selling point for rsnapshot is that it can generate a set of clones of the filesystem contents that are directly readable. That isn't as efficient as it can be, but it is very simple to work with, and it is done about as well as can be done with this sort of approach. Restoration basically requires no special tooling this way, so that is great if you want to restore from a generic rescue disk and not have to try to remember what commands to use. send-based tools for filesystems like brtrfs/zfs are SUPER-efficient in execution time/resources as they are filesystem-aware and don't need to stat everything on a filesystem to identify exactly what changed in an incremental backup. However, you're usually limited to restoring to another filesystem of the same type and have to use those tools. There are some scripts out there that automate the process of managing all of this (you need to maintain snapshots/etc to allow the incremental backups). There are a bunch of other tools for backing up specific applications/filesystems/etc as well. (Every database has one, which you should definitely use, and there are tools like volsync for k8s and so on.) Restic seems to be the most popular tool to backup to a small set of files on disk/cloud. I use duplicity for historical reasons, and restic does the same and probably supports more back-ends. These tools are very useful for cloud backups as they're very efficient about separating data/indexes and keeping local copies of the latter so you aren't paying to read back your archive data every time you do a new incremental backup, and they're very IO-efficient. Bacula is probably the best solution for tape backups of large numbers of systems, but it is really crufty and unwieldy. I would NOT use this to backup one host, and especially not to back up the host running bacula. Bootstrapping it is a pain. It is very much designed around a tape paradigm. If you have windows hosts you want to backup then be sure to find a solution that supports volume shadow copy - there aren't many. Bacula is one of them which is the main reason I even use it at this point. If you don't have that feature then you won't back up the registry, and you can imagine how losing that is on a windows machine. If you're just backing up documents though then anything will work, as long as files aren't open, because windows is extra-fussy about that. -- Rich
Re: [gentoo-user] Suggestions for backup scheme?
On Tuesday, 30 January 2024 18:15:09 GMT Grant Edwards wrote: > I need to set up some sort of automated backup on a couple Gentoo > machines (typical desktop software development and home use). One of > them used rsnapshot in the past but the crontab entries that drove > that have vanished :/ (presumably during a reinstall or upgrade -- > IIRC, it took a fair bit of trial and error to get the crontab entries > figured out). > > I believe rsnapshot ran nightly and kept daily snapshots for a week, > weekly snapshots for a month, and monthly snapshots for a couple > years. > > Are there other backup solutions that people would like to suggest I > look at to replace rsnapshot? I was happy enough with rsnapshot (when > it was running), but perhaps there's something else I should consider? > > -- > Grant You have probably seen the backup packages suggested in this wiki page? https://wiki.gentoo.org/wiki/Backup and what's available in the tree: https://packages.gentoo.org/categories/app-backup You may also want to consider integral filesystem solutions like btrfs and zfs, depending on your needs and how often your data change, as well as related scripts; e.g.: https://github.com/masc3d/btrfs-sxbackup signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Suggestions for backup scheme?
On 1/30/24 11:15, Grant Edwards wrote: I need to set up some sort of automated backup on a couple Gentoo machines (typical desktop software development and home use). One of them used rsnapshot in the past but the crontab entries that drove that have vanished :/ (presumably during a reinstall or upgrade -- IIRC, it took a fair bit of trial and error to get the crontab entries figured out). I believe rsnapshot ran nightly and kept daily snapshots for a week, weekly snapshots for a month, and monthly snapshots for a couple years. Are there other backup solutions that people would like to suggest I look at to replace rsnapshot? I was happy enough with rsnapshot (when it was running), but perhaps there's something else I should consider? -- Grant I backup, periodically: - corontab (user, root) - etc - hylafax daily: - data It all depend what you want you backup, how large is your data. For backup standard "rsync" over the network does the job OK I customized this rsync-bacup script: https://serverfault.com/questions/271527/explain-this-rsync-script-for-me-linux-backups # This script does personal backups to a rsync backup server. You will end up # with a 7 day rotating incremental backup. The incrementals will go # into subdirectories named after the day of the week, and the current # full backup goes into a directory called "current" # tri...@linuxcare.com
