Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
Default User writes: On Fri, 2023-09-01 at 23:15 +0200, Linux-Fan wrote: > Default User writes: [...] > > I HAVE used a number of other backup methodologies, including > > Borgbackup, for which I had high hopes, but was highly > > disappointed. > > Would you care to share in what regards BorgBackup failed you? [...] 2) I used borg for a while. Block-level de-duplication saved a ton of space. But . . . Per the borg documentation, I kept two separate (theoretically) identical backup sets, as repository 1 and repository 2, on backup drive A. Both repositories were daily backed up to backup drive B. Somehow, repository 1 on drive A got "messed up". I don't remember the details, and never determined why it happened. I had a copy of repository 1 on backup drive B, and two copies of repository 2 on backup drive B, so, no problem. I will just copy repository 1 on backup drive B to backup drive A. Right? Wrong. I could not figure out how to make that work, perhaps in part because of the way borg manages repositories by ID numbers, not by repository "names". [...] And, what is F/LOSS today can become closed and proprietary tomorrow, and thus unintentionally, or even deliberately, you and your data are trapped . . . (Audacity? CentOS?). So even though borg seems to be the flavor of the month, I decided no thanks. I think I'll just "Keep It Simple". Now if borg (or whatever) works for you, fine. Use it. This is just my explanation of why I looked elsewhere. YMMV. Thank you very much for taking the time to explain it detailedly. I can understand that corruption / messed up repositories are really one of the red flags for backup tools and hence a good reason to avoid such tools. Hence I can fully understand your decision. That there was no way to recover despite following the tools best practices (docs) does not improve things... Just a question for my understanding: You mentioned having multiple repositories. If I had the situation with two different repositories and one corrupted my first idea (if the backup program does not offer any internal functions for these purposes which you confirmed using the mailing list?) would be to copy the “good” repository at the file level (i.e. with rsync / tar whatever) and then afterwards update the copy to fixup any metadata that may be wrong. Did you try out this naive approach during your attempt for recovery? I think that currently I am not affected by such issues because I only keep the most recent state of the backup and do not have any history in my backups (beyond the “archive” which I keep separate and using my own program). Hence for me, indeed, the solution to re-create the repository in event of corruption is viable. But as the backup programs advertise the possibility to keep multiple states of the backup in one repository, it is indeed, essential, that one can “move around” such a repository on the file system while being able to continue adding to it even after swiching to a different/new location. I have never thought about testing such a use case for any of the tools that I tried, but I can see that it is actually quite the essential feature making it even more strange that it would not be available with Borg? TIA Linux-Fan öö pgp3s_4nB4ddI.pgp Description: PGP signature
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
On 9/5/23 17:39, Default User wrote:
On Tue, 2023-09-05 at 20:01 -0400, Default User wrote:
Now sudo du -sh / says that / seems to be using about 30 Gb. But sudo
du -sh /media/user/rsnapshot_backups_of_host, says that the backup
directory, /media/user/rsnapshot_backups_of_host on backup drive A,
is
using a whopping 88 Gb for 24 hourly, 7 daily, and 3 weekly!
That is better than (24+7+3) * 30 Gb = 1020 GB.
88 GB - 30 GB = 58 GB of churn over 24 hours, 7 days, and/or 3 weeks may
be reasonable for your workload. Are you doing multimedia content
creation? Databases? Disk imaging? Anything else big?
I am thinking, that CAN'T be right.
Maybe each hard link is being counted as a full, actual file, when
adding up the space allegedly used.
So, how can I determine how much space is really being used for the
backups?
AIUI 'rsync --link-dest' hard links files on the destination only when
both the file data and the file metadata are identical. If either
changes, 'rsync --link-dest' considers the files to be different and
does a transfer/ copy.
/var/log/* is a canonical degenerate example file-level deduplication.
My Debian daily driver /var/log is 83 MB. 34 copies of that is 1.8 GB.
The challenge is finding big files with slightly different content, big
files with identical content but different metadata, and/or large
numbers of files with either or both differences.
I would start by using jdupes(1) to find identical backup files on the
backup drive. Then use stat(1) or ls(1) on each group of files to find
different metadata. You may want to put the commands into scripts as
you figure them out.
To find files with mismatched content, I would use jdupes(1) with the
--partial-only option, then jdupes(1), stat(1), and/or ls(1) to check
data and metadata as above.
[BTW, the rsnapshot backups don't seem to take too much time, but
doing
rsync of external usb backup drive A to external usb backup drive B
does take over 90 minutes each time. And that's once a day, every
day!
Most of that time is apparently not for data transfer, but for rsync
building the indexes it needs each time.]
COW file systems such as ZFS provide a time vs. space caching trade-off.
Here is the command I use to rsync backup drive A
(/media/default/MSD1) to backup drive B
(/media/default/MSD2):
time sudo rsync -aAXHxvv --delete-after --numeric-ids --
info=progress2,stats2,name2 --
exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/med
ia
/*","/lost+found"} /media/default/MSD1/ /media/default/MSD2/
I do not use the --numeric-ids option. I use matching username/UID and
groupname/GID on all of my Debian and FreeBSD hosts. I want user/group
name translation on my Windows/Cygwin and macOS hosts.
Your -v, -v, and --info options are going to generate a lot of output.
I typically use the --progress and --stats options, and request more
only when trouble-shooting.
I do not use the --exclude option. If and when a system crashes, I want
everything; including files that an intruder may have placed in
locations that are commonly not backed up. This means that when using
the -x option, I must make sure to backup all file systems.
I just wanted to clarify:
Each time backup drive A is rsync'd to backup drive B, much more than
/media/user/MSD1/rsnapshot_backups_of_host is being rsync'd. All
of /media/user/MSD1 is being rsync'd, which is somewhere around
900Gb. Maybe that is why each rsync takes over 90 minutes!
Please run the above rsync(1) command, without -v -v --info, and with
--stats. Post your console session.
Use nmon(1) to watch the backup drives when doing the transfer. Tell us
what you see.
David
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
On Fri, 2023-09-01 at 23:15 +0200, Linux-Fan wrote: > Default User writes: > > > On Fri, 2023-09-01 at 07:25 -0500, John Hasler wrote: > > > Jason writes: > > > > Or how does your backup look like? > > See https://lists.debian.org/debian-user/2019/11/msg00073.html > and https://lists.debian.org/debian-user/2019/11/msg00420.html > > > > Just rsync. > > > > > > Sorry, I just couldn't resist chiming in here. > > > > I have never used OpenMediaVault. > > > > I HAVE used a number of other backup methodologies, including > > Borgbackup, for which I had high hopes, but was highly > > disappointed. > > Would you care to share in what regards BorgBackup failed you? > > I am currently using `bupstash` (not in Debian unfortunatly) and > `jmbb` > (which I wrote for myself in 2013) in parallel and am considering > switching > to `bupstash` which provides just about all features that I need. > > Here are my notes on these programs: > * https://masysma.net/37/backup_tests_borg_bupstash_kopia.xhtml > * https://masysma.net/32/jmbb.xhtml > > And also the Bupstash home page: > * https://bupstash.io/ > > IMHO borg is about the best backup program that you can get from the > Debian > repositories (if you need any of the modern features that is). The > only > issue I really had with it is that it was too slow for my use cases. > > > In the end, I currently have settled upon using rsnapshot to back > > up my > > single-machine, single-user setup to external external usb hard > > drive > > A, which is then copied to external usb hard drive B, using rsync. > > If > > you can do rsync, you can do rsnapshot. > > > > It's easy, especially when it comes to restoring, verifying, and > > impromptu access to data, to use random stuff, or even to just > > "check > > on" your data occasionally, to reassure yourself that it is still > > there. > > > > Yes, it does require considerable space (no data de-duplication), > > and > > the rsync of the backup drives does take considerable time. But to > > me, > > it is worth it, to avoid the methodological equivalent of "vendor > > lock- > > in". > > Yes, the “vendor lock-in” is really a thing especially when it comes > to > restoring a backup but the fancy backup software just does not > compile for > the platform or is not available for other reasons or you are stuck > on a > Windows laptop without Admin permissions (wost case scenario?). > > I mitigated this with `jmbb` by providing for a way to restore > individual > files also using third-party utilities and I intend to mitigate this > for > `bupstash` by writing my own restore program > (work-in progress: https://masysma.net/32/maxbupst.xhtml) > > > INB4: No, I don't do online backups. If people or organizations > > with > > nose problems want my data they are going to have to make at least > > a > > little effort to get it. And yes, I do know the 1-2-3 backup > > philosophy, which does seem like a good idea for many (most?) > > users. > > The problem I have with offline backups that it is an inconvenience > to carry > around copies and that this means they are always more out of date > than I > want them to be. Hence I rely on encryption to store backups on > untrusted > storages. > > [...] > > Short but comprehensive resource on the subject (includes some > advertising / > I am not affiliated / maybe this has outlived the product it > advertises for?): > http://www.taobackup.com/index.html > > YMMV > Linux-Fan > > öö 1) I did read http://www.taobackup.com/index.html a while back. It is an ad, but funny and worth the read nevertheless. 2) I used borg for a while. Block-level de-duplication saved a ton of space. But . . . Per the borg documentation, I kept two separate (theoretically) identical backup sets, as repository 1 and repository 2, on backup drive A. Both repositories were daily backed up to backup drive B. Somehow, repository 1 on drive A got "messed up". I don't remember the details, and never determined why it happened. I had a copy of repository 1 on backup drive B, and two copies of repository 2 on backup drive B, so, no problem. I will just copy repository 1 on backup drive B to backup drive A. Right? Wrong. I could not figure out how to make that work, perhaps in part because of the way borg manages repositories by ID numbers, not by repository "names". So, I posted the problem to the borg mailing list. And my jaw hit the floor. I was astounded that no one there was able to say that it is even possible to do such a repository copy/restore, let alone how to do so! The best advice was to just create a new repository and start a new backup series, starting now and going forward. For exmple, let's say the original series started 2022-01-01. And the repository A "problem" started 2023-01-01. If I just created a new repository A, starting with a new series beginning with 2023-01-01, then I only have one copy of the data from 2022, in repository B. If
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
On Tue, 2023-09-05 at 20:01 -0400, Default User wrote: > On Fri, 2023-09-01 at 22:44 +0200, Michel Verdier wrote: > > > > > On 2023-09-01, Default User wrote: > > > > > > > > > > > > > > Yes, it does require considerable space (no data > > > > > > > > > de-duplication), > > > > > > > > > and > > > > > > > > > the rsync of the backup drives does take considerable > > > > > > > > > time. > > > > > > > > > But to > > > > > > > > > me, > > > > > > > > > it is worth it, to avoid the methodological > > > > > > > > > equivalent > > > > > > > > > of > > > > > > > > > "vendor > > > > > > > > > lock- > > > > > > > > > in". > > > > > > > > > > You must have a bad configuration : rsnaphot de-duplicate > > > > > using > > > > > hard > > > > > links so you never have duplicated files. Keeping 52 weekly > > > > > and > > > > > 7 > > > > > daily > > > > > and 24 hourly I need only 130% of original space. And it > > > > > takes > > > > > minimal > > > > > time as it transfers only changes and can use ssh > > > > > compression. > > > > > > > > Okay, first: > > I said that my "system" files (everything except /home/[user] was > backed up using Timeshift. That is correct. It is done by Timeshift > automatically, once a day, as well as weekly, monthly, and yearly. > > But I was wrong about rsnapshot. I said that it was set up to back up > /home/[user] only. That is not correct. I now realize that I have it > set up to back up all of /, except for: > exclude /dev/* > exclude /proc/* > exclude /sys/* > exclude /tmp/* > exclude /run/* > exclude /mnt/* > exclude /media/* > exclude /lost+found > exclude /home/lost+found > exclude /var/lost+found > > Now sudo du -sh / says that / seems to be using about 30 Gb. But sudo > du -sh /media/user/rsnapshot_backups_of_host, says that the backup > directory, /media/user/rsnapshot_backups_of_host on backup drive A, > is > using a whopping 88 Gb for 24 hourly, 7 daily, and 3 weekly! > > I am thinking, that CAN'T be right. > Maybe each hard link is being counted as a full, actual file, when > adding up the space allegedly used. > > So, how can I determine how much space is really being used for the > backups? > > > > > - > -- > > > FWIW, here is my /etc/rsnapshot.conf file: > # > # rsnapshot.conf - rsnapshot configuration file # > # > # # > # PLEASE BE AWARE OF THE FOLLOWING RULE: # > # # > # This file requires tabs between elements # > # # > # > > ### > # CONFIG FILE VERSION # > ### > > config_version 1.2 > > ### > # SNAPSHOT ROOT DIRECTORY # > ### > > # All snapshots will be stored under this root directory. > # > snapshot_root /media/user/MSD1/rsnapshot_backups_of_host/ > > # If no_create_root is enabled, rsnapshot will not automatically > create > the > # snapshot_root directory. This is particularly useful if you are > backing > # up to removable media, such as a FireWire or USB drive. > # > no_create_root 1 > > # > # EXTERNAL PROGRAM DEPENDENCIES # > # > > # LINUX USERS: Be sure to uncomment "cmd_cp". This gives you extra > features. > # EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility. > # > # See the README file or the man page for more details. > # > cmd_cp /bin/cp > > # uncomment this to use the rm program instead of the built-in perl > routine. > # > cmd_rm /bin/rm > > # rsync must be enabled for anything to work. This is the only > command > that > # must be enabled. > # > cmd_rsync /usr/bin/rsync > > # Uncomment this to enable remote ssh backups over rsync. > # > #cmd_ssh /usr/bin/ssh > > # Comment this out to disable syslog support. > # > cmd_logger /usr/bin/logger > > # Uncomment this to specify the path to "du" for disk usage checks. > # If you have an older version of "du", you may also want to check > the > # "du_args" parameter below. > # > cmd_du /usr/bin/du > > # Uncomment this to specify the path to rsnapshot-diff. > # > cmd_rsnapshot_diff /usr/bin/rsnapshot-diff > > # Specify the path to a script (and any optional arguments) to run > right > # before rsnapshot syncs files > # > #cmd_preexec /path/to/preexec/script > > # Specify the path to a script (and any optional arguments) to run > right > # after rsnapshot syncs files > # > #cmd_postexec /path/to/postexec/script > > # Paths to lvcreate, lvremove, mount and umount commands, for use > with > # Linux LVMs. > # > #linux_lvm_cmd_lvcreate /sbin/lvcreate > #linux_lvm_cmd_lvremove /sbin/lvremove > #linux_lvm_cmd_mount /bin/mount > #linux_lvm_cmd_umount /bin/umount > > # > # BACKUP LEVELS /
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
On Fri, 2023-09-01 at 22:44 +0200, Michel Verdier wrote: > > > > On 2023-09-01, Default User wrote: > > > > > > > > > > > > Yes, it does require considerable space (no data > > > > > > > > de-duplication), > > > > > > > > and > > > > > > > > the rsync of the backup drives does take considerable > > > > > > > > time. > > > > > > > > But to > > > > > > > > me, > > > > > > > > it is worth it, to avoid the methodological equivalent > > > > > > > > of > > > > > > > > "vendor > > > > > > > > lock- > > > > > > > > in". > > > > > > > > You must have a bad configuration : rsnaphot de-duplicate using > > > > hard > > > > links so you never have duplicated files. Keeping 52 weekly and > > > > 7 > > > > daily > > > > and 24 hourly I need only 130% of original space. And it takes > > > > minimal > > > > time as it transfers only changes and can use ssh compression. > > > > Okay, first: I said that my "system" files (everything except /home/[user] was backed up using Timeshift. That is correct. It is done by Timeshift automatically, once a day, as well as weekly, monthly, and yearly. But I was wrong about rsnapshot. I said that it was set up to back up /home/[user] only. That is not correct. I now realize that I have it set up to back up all of /, except for: exclude /dev/* exclude /proc/* exclude /sys/* exclude /tmp/* exclude /run/* exclude /mnt/* exclude /media/* exclude /lost+found exclude /home/lost+found exclude /var/lost+found Now sudo du -sh / says that / seems to be using about 30 Gb. But sudo du -sh /media/user/rsnapshot_backups_of_host, says that the backup directory, /media/user/rsnapshot_backups_of_host on backup drive A, is using a whopping 88 Gb for 24 hourly, 7 daily, and 3 weekly! I am thinking, that CAN'T be right. Maybe each hard link is being counted as a full, actual file, when adding up the space allegedly used. So, how can I determine how much space is really being used for the backups? --- FWIW, here is my /etc/rsnapshot.conf file: # # rsnapshot.conf - rsnapshot configuration file # # # # # PLEASE BE AWARE OF THE FOLLOWING RULE: # # # # This file requires tabs between elements # # # # ### # CONFIG FILE VERSION # ### config_version 1.2 ### # SNAPSHOT ROOT DIRECTORY # ### # All snapshots will be stored under this root directory. # snapshot_root /media/user/MSD1/rsnapshot_backups_of_host/ # If no_create_root is enabled, rsnapshot will not automatically create the # snapshot_root directory. This is particularly useful if you are backing # up to removable media, such as a FireWire or USB drive. # no_create_root 1 # # EXTERNAL PROGRAM DEPENDENCIES # # # LINUX USERS: Be sure to uncomment "cmd_cp". This gives you extra features. # EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility. # # See the README file or the man page for more details. # cmd_cp /bin/cp # uncomment this to use the rm program instead of the built-in perl routine. # cmd_rm /bin/rm # rsync must be enabled for anything to work. This is the only command that # must be enabled. # cmd_rsync /usr/bin/rsync # Uncomment this to enable remote ssh backups over rsync. # #cmd_ssh /usr/bin/ssh # Comment this out to disable syslog support. # cmd_logger /usr/bin/logger # Uncomment this to specify the path to "du" for disk usage checks. # If you have an older version of "du", you may also want to check the # "du_args" parameter below. # cmd_du /usr/bin/du # Uncomment this to specify the path to rsnapshot-diff. # cmd_rsnapshot_diff /usr/bin/rsnapshot-diff # Specify the path to a script (and any optional arguments) to run right # before rsnapshot syncs files # #cmd_preexec /path/to/preexec/script # Specify the path to a script (and any optional arguments) to run right # after rsnapshot syncs files # #cmd_postexec /path/to/postexec/script # Paths to lvcreate, lvremove, mount and umount commands, for use with # Linux LVMs. # #linux_lvm_cmd_lvcreate /sbin/lvcreate #linux_lvm_cmd_lvremove /sbin/lvremove #linux_lvm_cmd_mount /bin/mount #linux_lvm_cmd_umount /bin/umount # # BACKUP LEVELS / INTERVALS # # Must be unique and in ascending order # # e.g. alpha, beta, gamma, etc. # # # retain hourly 24 retain daily 7 retain weekly 4 retain monthly 12 retain yearly 100 # GLOBAL OPTIONS # # All are optional, with sensible defaults # # Verbose level, 1 through 5. # 1 Quiet Print fatal errors only # 2 Default
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
> On 3 Sep 2023, at 19:16, David wrote: [...] > I have a Dell R320 fitted with 8 1T SAS drives, the hardware raid is > turned off as OpenMediaVault uses sorfware RAID. > If I turn the hardware raid on can I use Debian as the opperating > system? Hi David, In general, outside of certain relatively niche use cases, I believe software raid is to be preferred due to comparable performance and lack of future hardware compatibility/availability issues. See for example (if with some contradictions) https://www.techtarget.com/searchstorage/tip/Key-differences-in-software-RAID-vs-hardware-RAID Debian supports software raid, in the form of MDRAID. I seemm to recall this is usually combined with LVM. OpenZFS may be of interest, possibly root on ZFS too. https://en.m.wikipedia.org/wiki/OpenZFS https://openzfs.org/wiki/System_Administration https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/index.html There doesn't seem to be documentation for root on zfs for bookworm yet. I imagine the bullseye instructions, suitably adapted for repositories etc, might suffice, but I upgraded a root on ZFS installation from Buster to Bullseye to Bookworm (following the release notes in each case) so haven't tried to install from scratch in years. You will need to turn hardware raid off in either case (MDRAID or ZFS) If you have actual raid controller cards, that may not be possible iiuc - I'm sure someone (maybe even me) could advise if you provide details of raid hardware if relevant. Best wishes, Gareth > > Thank you for any help, > > David.
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
On Sat, 2023-09-02 at 23:57 +0200, Linux-Fan wrote: > Michael Kjörling writes: > > [...] > > > The biggest issue for me is ensuring that I am not dependent on > > _anything_ on the backed-up system itself to start restoring that > > system from a backup. In other words, enabling bare-metal > > restoration. > > I figure that I can always download a Debian live ISO, put that on > > a > > USB stick, set up an environment to access the (encrypted) backup > > drive, set up partitions on new disks, and start copying; if I were > > using backup software that uses some kind of custom format, that > > would > > include keeping a copy of an installation package of that and > > whatever > > else it needs for installing and running within a particular > > distribution version, and making sure to specifically test that, > > ideally without Internet access, so that I can get to the point of > > starting to copy things back. (I figure that the boot loader is the > > easy part to all this.) > > [...] > > My personal way to approach this is as follows: > > * I identify the material needed to restore. > It consists of > > - the backup itself > - suitable Linux OS to run a restore process on > - the backup software > - the backup key > - a password to decrypt the backup key > > * I create a live DVD (using `live-build`) containing > the Linux OS (including GUI, gparted and debian-installer!), > backup software (readily installed inside the live system), > backup key (as an encrypted file) but not the password nor > the backup itself. > > Instead I decided to add: > > - a copy of an SSH identity I can use to access a > read-only copy of the backup through my server and > - a copy of the encrypted password manager database > in case I forgot the backup password but not the > password manager password and also in case I would > be stuck with the Live DVD but not a copy of the > password such that I could use one of the password > manager passwords to access an online copy of the > backup. > > * When I still used physical media in my backup strategy > these were external SSDs (not ideal in terms of data > retention, I know). I partitioned them and made them > able to boot the customized live system (through syslinux). > > If you took such a drive and a PC of matching architecture > (say: amd64) then everything was in place to restore from > that drive (except for the password...). The resulting Debian > would probably be one release behind (because I rarely updated > the live image on the drive) but the data would be as up to > date as the contained backup. The assumtion here was that one > would be permitted to boot a custom OS off the drive or have > access to a Linux that could read it because I formatted the > “data” part with ext4 which is not natively readable on > Windows. > > In addition to that, each copy of my backups includes a copy of the > backup > program executable (a JAR file and a statically compiled Rust program > in my > case) and some Windows exe files that could be used to restore the > backup on > Windows machines in event of being stuck with a copy of the backup > “only”. > > While this scheme is pretty strong in theory, I update and test it > far too > rarely since it is not really easy to script the process, but at > least I > tested the correct working of the backup restore after creation of > the live > image by starting the restore from inside a VM. > > HTH > Linux-Fan > > öö > I have also been trying UpenMediaVault and it's an overkill for me. I have a Dell R320 fitted with 8 1T SAS drives, the hardware raid is turned off as OpenMediaVault uses sorfware RAID. If I turn the hardware raid on can I use Debian as the opperating system? Thank you for any help, David.
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
Michael Kjörling writes: [...] The biggest issue for me is ensuring that I am not dependent on _anything_ on the backed-up system itself to start restoring that system from a backup. In other words, enabling bare-metal restoration. I figure that I can always download a Debian live ISO, put that on a USB stick, set up an environment to access the (encrypted) backup drive, set up partitions on new disks, and start copying; if I were using backup software that uses some kind of custom format, that would include keeping a copy of an installation package of that and whatever else it needs for installing and running within a particular distribution version, and making sure to specifically test that, ideally without Internet access, so that I can get to the point of starting to copy things back. (I figure that the boot loader is the easy part to all this.) [...] My personal way to approach this is as follows: * I identify the material needed to restore. It consists of - the backup itself - suitable Linux OS to run a restore process on - the backup software - the backup key - a password to decrypt the backup key * I create a live DVD (using `live-build`) containing the Linux OS (including GUI, gparted and debian-installer!), backup software (readily installed inside the live system), backup key (as an encrypted file) but not the password nor the backup itself. Instead I decided to add: - a copy of an SSH identity I can use to access a read-only copy of the backup through my server and - a copy of the encrypted password manager database in case I forgot the backup password but not the password manager password and also in case I would be stuck with the Live DVD but not a copy of the password such that I could use one of the password manager passwords to access an online copy of the backup. * When I still used physical media in my backup strategy these were external SSDs (not ideal in terms of data retention, I know). I partitioned them and made them able to boot the customized live system (through syslinux). If you took such a drive and a PC of matching architecture (say: amd64) then everything was in place to restore from that drive (except for the password...). The resulting Debian would probably be one release behind (because I rarely updated the live image on the drive) but the data would be as up to date as the contained backup. The assumtion here was that one would be permitted to boot a custom OS off the drive or have access to a Linux that could read it because I formatted the “data” part with ext4 which is not natively readable on Windows. In addition to that, each copy of my backups includes a copy of the backup program executable (a JAR file and a statically compiled Rust program in my case) and some Windows exe files that could be used to restore the backup on Windows machines in event of being stuck with a copy of the backup “only”. While this scheme is pretty strong in theory, I update and test it far too rarely since it is not really easy to script the process, but at least I tested the correct working of the backup restore after creation of the live image by starting the restore from inside a VM. HTH Linux-Fan öö pgpjGcE1hDBcf.pgp Description: PGP signature
Backup systems (was: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq)
> More accurately, rsnapshot (which is basically a frontend to rsync) > tells rsync to do that; IIRC by passing --link-dest pointing at the > previous backup target directory. I've used a similar (tho hand-cooked) script running `rsync`. I switched to Bup a few years ago and saw a significant reduction in the size of my backups that is partly due to the deduplication *between* machines (I backup several Debian machines to the same backup repository) as well as because the deduplication occurs even when I move files around (most obvious when I move directories filled with large files like videos or music). Another advantage I've notice with Bup is that my backups are done much more quickly (because more of the work is done locally on the client machines which have SSDs, there's a lot less network communication, and much less work is done on the backup server which is a low-power machine with spinning rust). The great advantage of `rsync/rsnapshot` is that your backup is really "live": you can go `grep` and `find` through it with ease. With Bup you can theoretically do that as well (via `bup fuse`) but the performance is significantly lower :-( Stefan
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
On 2 Sep 2023 00:04 +0200, from mv...@free.fr (Michel Verdier): > rsnapshot use hard links on the backup filesystem. More accurately, rsnapshot (which is basically a frontend to rsync) tells rsync to do that; IIRC by passing --link-dest pointing at the previous backup target directory. And this is not an argument against rsnapshot/rsync; I use the combination myself, plus a home-grown script to prune old backups based on the amount of free space remaining on the backup disks rather than a fixed backup count. The one big downside of rsnapshot + rsync at least for me is that it has no real concept of whether a backup run completed or was aborted; the latter, for example, due to a system shutdown or out-of-memory condition while it's running. That really shouldn't happen often or even at all, but I've had it happen a few times over many years, and it's a bit of a pain when it does happen because you pretty much have to go in and delete the most recent backup and then renumber all the previous ones to get back into a sane state on the backup target. Yes, that can be added with another piece of a wrapper script, and I have on occasion contemplated doing just that; but it happens sufficiently rarely, and is noisy enough when it does happen, that it isn't really worth the effort in my particular situation. The biggest issue for me is ensuring that I am not dependent on _anything_ on the backed-up system itself to start restoring that system from a backup. In other words, enabling bare-metal restoration. I figure that I can always download a Debian live ISO, put that on a USB stick, set up an environment to access the (encrypted) backup drive, set up partitions on new disks, and start copying; if I were using backup software that uses some kind of custom format, that would include keeping a copy of an installation package of that and whatever else it needs for installing and running within a particular distribution version, and making sure to specifically test that, ideally without Internet access, so that I can get to the point of starting to copy things back. (I figure that the boot loader is the easy part to all this.) -- Michael Kjörling https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
Michel Verdier writes: On 2023-09-01, Default User wrote: > Yes, it does require considerable space (no data de-duplication), and > the rsync of the backup drives does take considerable time. But to me, > it is worth it, to avoid the methodological equivalent of "vendor lock- > in". You must have a bad configuration : rsnaphot de-duplicate using hard links so you never have duplicated files. Keeping 52 weekly and 7 daily and 24 hourly I need only 130% of original space. And it takes minimal time as it transfers only changes and can use ssh compression. It highly depends on the type of data that is being backed up. For my regular user files, I think a file-based deduplication works OK. But for my VM images, hardlinks would only save space for those VMs which did not run between the current and the preceding backup. Btw.: I am personally not using any hard-link based approach, mostly due to the missing encryption and integrity protection of data and metadata. HTH Linux-Fan öö pgp_XdT1JGgxv.pgp Description: PGP signature
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
Default User writes: On Fri, 2023-09-01 at 07:25 -0500, John Hasler wrote: > Jason writes: > > Or how does your backup look like? See https://lists.debian.org/debian-user/2019/11/msg00073.html and https://lists.debian.org/debian-user/2019/11/msg00420.html > Just rsync. Sorry, I just couldn't resist chiming in here. I have never used OpenMediaVault. I HAVE used a number of other backup methodologies, including Borgbackup, for which I had high hopes, but was highly disappointed. Would you care to share in what regards BorgBackup failed you? I am currently using `bupstash` (not in Debian unfortunatly) and `jmbb` (which I wrote for myself in 2013) in parallel and am considering switching to `bupstash` which provides just about all features that I need. Here are my notes on these programs: * https://masysma.net/37/backup_tests_borg_bupstash_kopia.xhtml * https://masysma.net/32/jmbb.xhtml And also the Bupstash home page: * https://bupstash.io/ IMHO borg is about the best backup program that you can get from the Debian repositories (if you need any of the modern features that is). The only issue I really had with it is that it was too slow for my use cases. In the end, I currently have settled upon using rsnapshot to back up my single-machine, single-user setup to external external usb hard drive A, which is then copied to external usb hard drive B, using rsync. If you can do rsync, you can do rsnapshot. It's easy, especially when it comes to restoring, verifying, and impromptu access to data, to use random stuff, or even to just "check on" your data occasionally, to reassure yourself that it is still there. Yes, it does require considerable space (no data de-duplication), and the rsync of the backup drives does take considerable time. But to me, it is worth it, to avoid the methodological equivalent of "vendor lock- in". Yes, the “vendor lock-in” is really a thing especially when it comes to restoring a backup but the fancy backup software just does not compile for the platform or is not available for other reasons or you are stuck on a Windows laptop without Admin permissions (wost case scenario?). I mitigated this with `jmbb` by providing for a way to restore individual files also using third-party utilities and I intend to mitigate this for `bupstash` by writing my own restore program (work-in progress: https://masysma.net/32/maxbupst.xhtml) INB4: No, I don't do online backups. If people or organizations with nose problems want my data they are going to have to make at least a little effort to get it. And yes, I do know the 1-2-3 backup philosophy, which does seem like a good idea for many (most?) users. The problem I have with offline backups that it is an inconvenience to carry around copies and that this means they are always more out of date than I want them to be. Hence I rely on encryption to store backups on untrusted storages. [...] Short but comprehensive resource on the subject (includes some advertising / I am not affiliated / maybe this has outlived the product it advertises for?): http://www.taobackup.com/index.html YMMV Linux-Fan öö pgptt2qvn0tQe.pgp Description: PGP signature
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
On Fri, 2023-09-01 at 07:25 -0500, John Hasler wrote: > Jason writes: > > Or how does your backup look like? > > Just rsync. Sorry, I just couldn't resist chiming in here. I have never used OpenMediaVault. I HAVE used a number of other backup methodologies, including Borgbackup, for which I had high hopes, but was highly disappointed. In the end, I currently have settled upon using rsnapshot to back up my single-machine, single-user setup to external external usb hard drive A, which is then copied to external usb hard drive B, using rsync. If you can do rsync, you can do rsnapshot. It's easy, especially when it comes to restoring, verifying, and impromptu access to data, to use random stuff, or even to just "check on" your data occasionally, to reassure yourself that it is still there. Yes, it does require considerable space (no data de-duplication), and the rsync of the backup drives does take considerable time. But to me, it is worth it, to avoid the methodological equivalent of "vendor lock- in". INB4: No, I don't do online backups. If people or organizations with nose problems want my data they are going to have to make at least a little effort to get it. And yes, I do know the 1-2-3 backup philosophy, which does seem like a good idea for many (most?) users. Also, I really like Clonezilla for occasional brute-force, scorched- earth backups, such as when preparing for a complete reinstall or release upgrade, or when switching to a new computer. (Note: everything above applies to backing up my data: (/home/[user] only). For backing up the actual system: (/ except for /home/[user]), I use (and like) Timeshift. It has saved my donkey more than once!) Well, that's what works for me. Feel free to disregard this. It may not work for you. And, "if it breaks, you get to keep both pieces"!
Re: I uninstalled OpenMediaVault (because totally overkill for me) and replaced it with borgbackup and rsyncq
Jason writes: > Or how does your backup look like? Just rsync. -- John Hasler j...@sugarbit.com Elmwood, WI USA
