Re: [BackupPC-users] Backup Share Content
On 7/29/23 21:11, backu...@kosowsky.org wrote: To be clear, you don't really need to add '--delete' as it is one of the *default* parameters and the comments in config.pl even warns you not to do so. Rather, please don't delete parameters that you don't understand or intend :) Just to reiterate it was my own mistakes that caused the issue "for some ancient reason I had per client overrides of the rsyncArgs that were the cause of my issues." I don't know why I had these overrides in the client configs. They must have been there since 2008 when I first set up this complex backup environment then. I did a lot of searches for example set ups back then to get ssh working, disks recognized correctly etc... and I may have followed an un-official guide that led me to the config overrides I have. In all those years I never had a situation where I needed to restore a whole share of data till recently. BackupPC has run reliably in its misconfiguration for all of that time. Now having run a series of backups with the deletion of my overrides all is as expected for the share content that I've now taken a closer look at. I'm sorry that in my panic my statements appeared to blame the program. -- Jim KR ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Backup Share Content
To be clear, you don't really need to add '--delete' as it is one of the *default* parameters and the comments in config.pl even warns you not to do so. Rather, please don't delete parameters that you don't understand or intend :) backuppc--- via BackupPC-users wrote at about 21:00:25 +0300 on Saturday, July 29, 2023: > jbk wrote: > > > I'm wondering if I need to add a "delete" argument to the > > rsync line but am not sure of the consequences. > > Yes, you do need to add --delete. See e.g. here for the default rsync > parameters: > https://github.com/backuppc/backuppc/blob/master/conf/config.pl#L1288 > > BR, > Fabian > > > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:https://github.com/backuppc/backuppc/wiki > Project: https://backuppc.github.io/backuppc/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Backup Share Content
You are right, jbk is mistaken. It boggles my mind that often people's first reaction to an error is to assume that a stable and well-tested program is fundamentally broken (e.g., doesn't do actual backups but rather just adds and doesn't delete files). The first, most appropriate, and most humble response should be "what am I doing wrong?" or "why don't I understand how the program works?" I can assure you that in my experience, user error (as is the case here) is the far more likely situation... Paul Fox wrote at about 11:50:17 -0400 on Saturday, July 29, 2023: > jbk wrote: > >BackupPC-4.4.0-9.el9.x86_64 > > > >I see now this is a bigger problem than I thought initially. I thought > >that each backup # was a restore point to the state of the share at the > >time of that backup but instead it is an accumulation of all backups > w/o > >reflecting any of the deletions of files now non existent on the source > > Surely that's not correct. If I do a restore of a specific backup, > I'd expect to get exactly the files that were present when that backup > was run. If I wanted files that had been deleted before that backup > was run, I would understand that I'd need to restore them from a > previous backup. > > If this isn't the case, then I'm... shocked, I guess. > > paul > =-- > paul fox, p...@foxharp.boston.ma.us (arlington, ma, where it's 85.1 degrees) > > > > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:https://github.com/backuppc/backuppc/wiki > Project: https://backuppc.github.io/backuppc/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Backup Share Content
Before casting mistaken aspersions about how backuppc works, why did
you delete the '--delete' line in the default rsync args?
>From stock config.pl for backuppc 4.x
NOTE how the comment says "DO NOT EDIT (emphasis added) the first set
unless you have a good understanding of rsync options.
#
# Arguments to rsync for backup. Do not edit the first set unless you
# have a good understanding of rsync options.
#
$Conf{RsyncArgs} = [
'--super',
'--recursive',
'--protect-args',
'--numeric-ids',
'--perms',
'--owner',
'--group',
'-D',
'--times',
'--links',
'--hard-links',
'--delete',
'--delete-excluded',
'--one-file-system',
'--partial',
'--log-format=log: %o %i %B %8U,%8G %9l %f%L',
'--stats',
#
# Add additional arguments here, for example --acls or --xattrs
# if all the clients support them.
#
#'--acls',
#'--xattrs',
];
jbk wrote at about 10:08:30 -0400 on Saturday, July 29, 2023:
> BackupPC-4.4.0-9.el9.x86_64
>
> I see now this is a bigger problem than I thought initially.
> I thought that each backup # was a restore point to the
> state of the share at the time of that backup but instead it
> is an accumulation of all backups w/o reflecting any of the
> deletions of files now non existent on the source machine. I
> had never browsed the backups to see if they were what I
> expected for one to one state of the current source. I've
> always used the default rsync args and now I think I need to
> modify them but I guess I'm really nave as to how backuppc
> is working.
> Some historical background. I've been using BackupPC for at
> least 15 years and migrated my backups from version 3. to
> version 4 a year and a half ago when I installed Rocky8 but
> I'm now running Rocky9 on the server. My logs, error and
> otherwise are not showing any backup issues.
>
> Here are a few lines from a recent backup of one of my
> machines xferlogs to illustrate the current rsync arguments
> in use. The full list of includes/excludes has been
> truncated. Note that the pasted lines don't wrap, at least
> not for me in my email client.
>
> #
>
> XferLOG file /var/lib/BackupPC//pc/xjbk7/XferLOG.64.z created 2023-07-25
> 15:45:31
> Backup prep: type = full, case = 3, inPlace = 1, doDuplicate = 1, newBkupNum
> = 64, newBkupIdx = 3, lastBkupNum = , lastBkupIdx = (FillCycle = 0,
> noFillCnt = 0)
> Executing /usr/share/BackupPC/bin/BackupPC_backupDuplicate -m -h xjbk7
> Xfer PIDs are now 540945
> Copying backup #63 to #64
> Xfer PIDs are now 540946,540945
> BackupPC_refCountUpdate: host xjbk7 got 0 errors (took 2 secs)
> Xfer PIDs are now 540945
> BackupPC_backupDuplicate: got 0 errors and 0 file open errors
> Finished BackupPC_backupDuplicate (running time: 7 sec)
> Running: /usr/bin/rsync_bpc --bpc-top-dir /var/lib/BackupPC/ --bpc-host-name
> xjbk7 --bpc-share-name / --bpc-bkup-num 64 --bpc-bkup-comp 3
> --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1 --bpc-bkup-inode0 121320
> --bpc-log-level 1 --bpc-attrib-new -e /usr/bin/ssh\ -l\ backuppc\ \
> --rsync-path=/usr/bin/sudo\ /usr/bin/rsync --numeric-ids --perms --owner
> --group -D --links --hard-links --times --block-size=2048 --recursive
> --one-file-system --checksum --timeout=72000 --include=/boot
>
> #
>
> I'm wondering if I need to add a "delete" argument to the
> rsync line but am not sure of the consequences.
>
> What I want is that for say backup #32 that a restore of
> that backup would be just the current state of the files on
> that machine at the time of backup and backup #31 would be a
> reflection of the share state at the time of that prior backup.
>
> jbk
>
> On 7/26/23 20:36, jbk wrote:
> > On one of my machine shares I did some file consolidation
> > that involved creating a new top level directory and
> > moving files from adjacent top level directories into sub
> > directories under the newly created directory I then
> > deleted those other empty top level directories.
> > Backuppc ran a couple days later a incremental backup, and
> > in this backup there was the new directory with sub
> > directories and files I had moved as well as the old and
> > no longer existing top level directories and files. I
> > waited till the next nightly ran and then checked again
> > and the deleted directories and files were still present.
> > So I haven't lost data but if I had a disk failure the
> > restore would recreate deleted data on the new destination
> > share. So the question is how do I delete old share data
> > structures in my backups or how should I approach such
> > changes so I don't end up with over populated restores.
> > --
> > Jim KR
>
>
> --
> Jim KR
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.ne
Re: [BackupPC-users] Backup Share Content
On 7/29/23 14:00, backu...@mailfri.com wrote: jbk wrote: I'm wondering if I need to add a "delete" argument to the rsync line but am not sure of the consequences. Yes, you do need to add --delete. See e.g. here for the default rsync parameters: https://github.com/backuppc/backuppc/blob/master/conf/config.pl#L1288 BR, Fabian Well for some ancient reason I had per client overrides of the rsyncArgs that were the cause of my issues. Just removed those overrides and ran new backups on all to confirm the corrections achieved the intended outcome and so far it is good. So thanks Fabian. -- Jim KR ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Backup Share Content
jbk wrote: > I'm wondering if I need to add a "delete" argument to the > rsync line but am not sure of the consequences. Yes, you do need to add --delete. See e.g. here for the default rsync parameters: https://github.com/backuppc/backuppc/blob/master/conf/config.pl#L1288 BR, Fabian ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Backup Share Content
Mine (4.4.0) works exactly as expected, i.e. each backup is a moment-in-time snapshot. There's clearly something amiss with jbk's setup. Robert Trevellyan On Sat, Jul 29, 2023 at 11:52 AM Paul Fox wrote: > jbk wrote: > >BackupPC-4.4.0-9.el9.x86_64 > > > >I see now this is a bigger problem than I thought initially. I > thought > >that each backup # was a restore point to the state of the share at > the > >time of that backup but instead it is an accumulation of all backups > w/o > >reflecting any of the deletions of files now non existent on the > source > > Surely that's not correct. If I do a restore of a specific backup, > I'd expect to get exactly the files that were present when that backup > was run. If I wanted files that had been deleted before that backup > was run, I would understand that I'd need to restore them from a > previous backup. > > If this isn't the case, then I'm... shocked, I guess. > > paul > =-- > paul fox, p...@foxharp.boston.ma.us (arlington, ma, where it's 85.1 > degrees) > > > > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:https://github.com/backuppc/backuppc/wiki > Project: https://backuppc.github.io/backuppc/ > ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Backup Share Content
jbk wrote: >BackupPC-4.4.0-9.el9.x86_64 > >I see now this is a bigger problem than I thought initially. I thought >that each backup # was a restore point to the state of the share at the >time of that backup but instead it is an accumulation of all backups w/o >reflecting any of the deletions of files now non existent on the source Surely that's not correct. If I do a restore of a specific backup, I'd expect to get exactly the files that were present when that backup was run. If I wanted files that had been deleted before that backup was run, I would understand that I'd need to restore them from a previous backup. If this isn't the case, then I'm... shocked, I guess. paul =-- paul fox, p...@foxharp.boston.ma.us (arlington, ma, where it's 85.1 degrees) ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Backup Share Content
BackupPC-4.4.0-9.el9.x86_64 I see now this is a bigger problem than I thought initially. I thought that each backup # was a restore point to the state of the share at the time of that backup but instead it is an accumulation of all backups w/o reflecting any of the deletions of files now non existent on the source machine. I had never browsed the backups to see if they were what I expected for one to one state of the current source. I've always used the default rsync args and now I think I need to modify them but I guess I'm really nave as to how backuppc is working. Some historical background. I've been using BackupPC for at least 15 years and migrated my backups from version 3. to version 4 a year and a half ago when I installed Rocky8 but I'm now running Rocky9 on the server. My logs, error and otherwise are not showing any backup issues. Here are a few lines from a recent backup of one of my machines xferlogs to illustrate the current rsync arguments in use. The full list of includes/excludes has been truncated. Note that the pasted lines don't wrap, at least not for me in my email client. # XferLOG file /var/lib/BackupPC//pc/xjbk7/XferLOG.64.z created 2023-07-25 15:45:31 Backup prep: type = full, case = 3, inPlace = 1, doDuplicate = 1, newBkupNum = 64, newBkupIdx = 3, lastBkupNum = , lastBkupIdx = (FillCycle = 0, noFillCnt = 0) Executing /usr/share/BackupPC/bin/BackupPC_backupDuplicate -m -h xjbk7 Xfer PIDs are now 540945 Copying backup #63 to #64 Xfer PIDs are now 540946,540945 BackupPC_refCountUpdate: host xjbk7 got 0 errors (took 2 secs) Xfer PIDs are now 540945 BackupPC_backupDuplicate: got 0 errors and 0 file open errors Finished BackupPC_backupDuplicate (running time: 7 sec) Running: /usr/bin/rsync_bpc --bpc-top-dir /var/lib/BackupPC/ --bpc-host-name xjbk7 --bpc-share-name / --bpc-bkup-num 64 --bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1 --bpc-bkup-inode0 121320 --bpc-log-level 1 --bpc-attrib-new -e /usr/bin/ssh\ -l\ backuppc\ \ --rsync-path=/usr/bin/sudo\ /usr/bin/rsync --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --one-file-system --checksum --timeout=72000 --include=/boot # I'm wondering if I need to add a "delete" argument to the rsync line but am not sure of the consequences. What I want is that for say backup #32 that a restore of that backup would be just the current state of the files on that machine at the time of backup and backup #31 would be a reflection of the share state at the time of that prior backup. jbk On 7/26/23 20:36, jbk wrote: On one of my machine shares I did some file consolidation that involved creating a new top level directory and moving files from adjacent top level directories into sub directories under the newly created directory I then deleted those other empty top level directories. Backuppc ran a couple days later a incremental backup, and in this backup there was the new directory with sub directories and files I had moved as well as the old and no longer existing top level directories and files. I waited till the next nightly ran and then checked again and the deleted directories and files were still present. So I haven't lost data but if I had a disk failure the restore would recreate deleted data on the new destination share. So the question is how do I delete old share data structures in my backups or how should I approach such changes so I don't end up with over populated restores. -- Jim KR -- Jim KR ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
