Re: [BackupPC-users] Testing full restore of backuppc... [private version]
Jeff,
The tar XferMethod doesn't capture acls and xattrs during backup.
Direct Restore in the CGI interface uses the XferMethod setting.
Craig
On Thu, May 21, 2020 at 10:05 PM wrote:
> I also assume that tar doesn't capture ACLs and XATTRs for backup
> either then
>
> What transfer mechanism does the CGI restore use?
> Because when I use the direct download mode, it also doesn't restore
> the ACLs and XATTRs.
>
> In any case, I guess I really need to figure out how to use rsync for
> restore...
>
>
> Craig Barratt via BackupPC-users wrote at about 21:50:40 -0700 on
> Thursday, May 21, 2020:
> > Jeff,
> >
> > Unfortunately BackupPC_tarCreate doesn't support acls. Over the years
> > different flavors of tar supported different archive formats for certain
> > extensions (eg, long file names etc). The POSIX standard for PAX
> headers
> > unified some of the those disparate formats, but didn't define acl or
> xattr
> > support.
> >
> > Over the last few years it does look like GNU tar provides support for
> > acls, but using PAX headers that are not standard. Looking at the tar
> > source, it uses headers like SCHILY.acl.access, SCHILY.xattr etc.
> > Supporting those headers appears to require the acls and xattrs to be
> > converted to descriptive strings. Currently BackupPC rsync treats acls
> and
> > xattr as binary blobs of data that it doesn't need to interpret. So
> > unfortunately it would be quit difficult to add acl and xattr support to
> > BackupPC_tarCreate.
> >
> > Craig
> >
> > On Tue, May 19, 2020 at 11:49 PM wrote:
> >
> > >
> > > Now that I have btrfs snapshots set up, I decided to test a full
> > > backup and restore by comparing the snapshot with the backup-restore
> > > via rsync, using the following command:
> > > sudo -u backuppc /usr/share/backuppc/bin/BackupPC_tarCreate -h
> myhost
> > > -n -1 -s myshare . | sudo tar --acls --selinux --xattrs -xvf -
> > >
> > > Interestingly, I found that everything worked *except* that it failed
> > > to copy any sockets or any extended attributes.
> > >
> > > 1. Sockets were not copied at all - but that is seemingly just a tar
> > >limitation since tar can't copy 'special' files.
> > >Indeed, backuppc-fuse shows that the files are actually backed up
> by
> > > bakcuppc
> > >
> > > 2. Extended attributes (ACLs and SELinux context) were *never*
> restored
> > >
> > >This seems to be a problem with 'BackupPC_tarCreate" since:
> > >a] Using tar alone, I can copy the files with all their extended
> > > attributes
> > > cd ; tar --acls --selinux --xattrs -cf - mac ) | tar
> xf -
> > >b] Similarly, raw rsync copies all the files faithfully
> > >rsync -navcxXAOH --delete .
> > >b] Backuppc-fuse shows the extended attributes
> > > (though that being said backuppc-fuse adds SELinux context
> attributes
> > > to files that don't have them... perhaps there is something
> wrong
> > > with the inheritance??
> > >
> > > Note: I tried adding ' --xargs --acls --selinux --xattrs'
> > > to $Conf{TarClientRestoreCmd} but that didn't help.
> > >
> > > So, 2 questions:
> > > 1. Why doesn't BackupPC_tarCreate restore the extended attributes?
> > > 2. Why does backuppc-fuse show extended attributes for files that
> > >don't have them originally?
> > >
> > > --
> > > Note: I am running ubuntu 18.04 with rsync 3.1.2 and backuppc 4.3.2
> > >
> > >
> > > ___
> > > BackupPC-users mailing list
> > > BackupPC-users@lists.sourceforge.net
> > > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > > Wiki:http://backuppc.wiki.sourceforge.net
> > > Project: http://backuppc.sourceforge.net/
> > >
> > ___
> > BackupPC-users mailing list
> > BackupPC-users@lists.sourceforge.net
> > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > Wiki:http://backuppc.wiki.sourceforge.net
> > Project: http://backuppc.sourceforge.net/
>
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Testing full restore of backuppc... [private version]
I also assume that tar doesn't capture ACLs and XATTRs for backup
either then
What transfer mechanism does the CGI restore use?
Because when I use the direct download mode, it also doesn't restore
the ACLs and XATTRs.
In any case, I guess I really need to figure out how to use rsync for restore...
Craig Barratt via BackupPC-users wrote at about 21:50:40 -0700 on Thursday, May
21, 2020:
> Jeff,
>
> Unfortunately BackupPC_tarCreate doesn't support acls. Over the years
> different flavors of tar supported different archive formats for certain
> extensions (eg, long file names etc). The POSIX standard for PAX headers
> unified some of the those disparate formats, but didn't define acl or xattr
> support.
>
> Over the last few years it does look like GNU tar provides support for
> acls, but using PAX headers that are not standard. Looking at the tar
> source, it uses headers like SCHILY.acl.access, SCHILY.xattr etc.
> Supporting those headers appears to require the acls and xattrs to be
> converted to descriptive strings. Currently BackupPC rsync treats acls and
> xattr as binary blobs of data that it doesn't need to interpret. So
> unfortunately it would be quit difficult to add acl and xattr support to
> BackupPC_tarCreate.
>
> Craig
>
> On Tue, May 19, 2020 at 11:49 PM wrote:
>
> >
> > Now that I have btrfs snapshots set up, I decided to test a full
> > backup and restore by comparing the snapshot with the backup-restore
> > via rsync, using the following command:
> > sudo -u backuppc /usr/share/backuppc/bin/BackupPC_tarCreate -h myhost
> > -n -1 -s myshare . | sudo tar --acls --selinux --xattrs -xvf -
> >
> > Interestingly, I found that everything worked *except* that it failed
> > to copy any sockets or any extended attributes.
> >
> > 1. Sockets were not copied at all - but that is seemingly just a tar
> >limitation since tar can't copy 'special' files.
> >Indeed, backuppc-fuse shows that the files are actually backed up by
> > bakcuppc
> >
> > 2. Extended attributes (ACLs and SELinux context) were *never* restored
> >
> >This seems to be a problem with 'BackupPC_tarCreate" since:
> >a] Using tar alone, I can copy the files with all their extended
> > attributes
> > cd ; tar --acls --selinux --xattrs -cf - mac ) | tar xf -
> >b] Similarly, raw rsync copies all the files faithfully
> >rsync -navcxXAOH --delete .
> >b] Backuppc-fuse shows the extended attributes
> > (though that being said backuppc-fuse adds SELinux context attributes
> > to files that don't have them... perhaps there is something wrong
> > with the inheritance??
> >
> > Note: I tried adding ' --xargs --acls --selinux --xattrs'
> > to $Conf{TarClientRestoreCmd} but that didn't help.
> >
> > So, 2 questions:
> > 1. Why doesn't BackupPC_tarCreate restore the extended attributes?
> > 2. Why does backuppc-fuse show extended attributes for files that
> >don't have them originally?
> >
> > --
> > Note: I am running ubuntu 18.04 with rsync 3.1.2 and backuppc 4.3.2
> >
> >
> > ___
> > BackupPC-users mailing list
> > BackupPC-users@lists.sourceforge.net
> > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > Wiki:http://backuppc.wiki.sourceforge.net
> > Project: http://backuppc.sourceforge.net/
> >
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Testing full restore of backuppc... [private version]
Jeff,
Unfortunately BackupPC_tarCreate doesn't support acls. Over the years
different flavors of tar supported different archive formats for certain
extensions (eg, long file names etc). The POSIX standard for PAX headers
unified some of the those disparate formats, but didn't define acl or xattr
support.
Over the last few years it does look like GNU tar provides support for
acls, but using PAX headers that are not standard. Looking at the tar
source, it uses headers like SCHILY.acl.access, SCHILY.xattr etc.
Supporting those headers appears to require the acls and xattrs to be
converted to descriptive strings. Currently BackupPC rsync treats acls and
xattr as binary blobs of data that it doesn't need to interpret. So
unfortunately it would be quit difficult to add acl and xattr support to
BackupPC_tarCreate.
Craig
On Tue, May 19, 2020 at 11:49 PM wrote:
>
> Now that I have btrfs snapshots set up, I decided to test a full
> backup and restore by comparing the snapshot with the backup-restore
> via rsync, using the following command:
> sudo -u backuppc /usr/share/backuppc/bin/BackupPC_tarCreate -h myhost
> -n -1 -s myshare . | sudo tar --acls --selinux --xattrs -xvf -
>
> Interestingly, I found that everything worked *except* that it failed
> to copy any sockets or any extended attributes.
>
> 1. Sockets were not copied at all - but that is seemingly just a tar
>limitation since tar can't copy 'special' files.
>Indeed, backuppc-fuse shows that the files are actually backed up by
> bakcuppc
>
> 2. Extended attributes (ACLs and SELinux context) were *never* restored
>
>This seems to be a problem with 'BackupPC_tarCreate" since:
>a] Using tar alone, I can copy the files with all their extended
> attributes
> cd ; tar --acls --selinux --xattrs -cf - mac ) | tar xf -
>b] Similarly, raw rsync copies all the files faithfully
>rsync -navcxXAOH --delete .
>b] Backuppc-fuse shows the extended attributes
> (though that being said backuppc-fuse adds SELinux context attributes
> to files that don't have them... perhaps there is something wrong
> with the inheritance??
>
> Note: I tried adding ' --xargs --acls --selinux --xattrs'
> to $Conf{TarClientRestoreCmd} but that didn't help.
>
> So, 2 questions:
> 1. Why doesn't BackupPC_tarCreate restore the extended attributes?
> 2. Why does backuppc-fuse show extended attributes for files that
>don't have them originally?
>
> --
> Note: I am running ubuntu 18.04 with rsync 3.1.2 and backuppc 4.3.2
>
>
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] List of All Files Being Backed Up
DW, I'd recommend using BackupPC_ls -R to recursively list all the files in a particular backup. You'd need to run it only each of your hosts on the latest backup. Craig On Tue, May 19, 2020 at 4:41 PM David Wynn via BackupPC-users < backuppc-users@lists.sourceforge.net> wrote: > I’ve tried to search using some different keywords and combinations but > have had no luck in finding an answer. Here’s my situation … I am backing > up around 150 different files from a NAS but using multiple hosts to do so > in order to keep the file transfer size down to <20GB at a time. This is > across the ‘internet’ so bandwidth is not under my control and I want to > make sure the jobs don’t crap out at bad times. I’ve found that <20GB at a > time usually works great. > > > > But, in order to manage the sizes I first created the individual lists > based on the size of the files/directories and input them manually into my > config files for each host. For example, HOST1 may have 40 smaller files > to get to the 20GB limit whereas HOST10 may only have 1 file/directory to > get to the limit. > > > > Now I have the problem of trying to find an individual file/directory from > around 18 different HOSTx setups. > > > > Is there an easy way to get/create a listing that would should the HOSTx > and the files/directories that are being backed up under it? I have > thought of trying to write a ‘script’ to traverse the individual HOSTx.pl > files and extract the info – but my scripting is purely of a W10 nature and > even at that is poor and wordy. (Oh for the days of COBOL and PL/1.) > > > > Just wondering if there is something I have missed in the documentation or > in trying to search the forums. Someone must have had this problem before > me. > > > > Thanks for your help > > > > DW > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] replication of data pool
Mike, The cpool isn't structured in a way that makes it possible to just copy recently backed-up files. Ged's suggestion (just run another BackupPC instance offsite) is worth considering. That also provides more robustness to certain failures (eg undetected filesystem corruption on the primary BackupPC server). Craig On Wed, May 20, 2020 at 12:37 PM Mike Hughes wrote: > Hi, we're currently syncing our cpool to an off-site location on a weekly > basis. Would it be feasible to only sync the latest of each backup rather > than the entire pool? > > To elaborate, on Saturdays we run an rsync of the entire cpool to another > server to provide disaster recovery options. Is it possible/reasonable to > just copy the data from the night before? Or, with de-duplication and > compression, would we really save much space/transfer time? If so, what is > the best way to grab just one night's worth of backups while still > preserving a full recovery? > > Just curious if someone is already doing this and how you sorted it out. > > Thanks! > Mike > > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Rsync restore from the command line
Jeff, Sure, that's possible. But given that it was necessary to use a file to pass along the list of files to restore, it seemed easier to just use that one mechanism. It would be pretty easy to write a wrapper script that takes command-line arguments and writes the request file and passes it along to BackupPC_restore ((see lib/BackupPC/CGI/Restore.pm) . Craig On Wed, May 20, 2020 at 11:59 AM wrote: > Couldn't their be an option to read the to-be-restored files from a > file (similar to what tar and rsync allow) but allowing basic restores > to be done form the command line. > Other parameters could either be via command line or added config.pl > settings if more permanent. > Craig Barratt via BackupPC-users wrote at about 11:31:37 -0700 on > Wednesday, May 20, 2020: > > Jeff, > > > > For restores, there could be a long list of specific files or > directories > > to restore, which might not fit on the command line, so that's what > > triggered putting everything in a request file and just passing its > name. > > There are also several other settings specific to the restore (eg, the > path > > to restore to etc), none of which are in config files. > > > > Craig > > > > On Wed, May 20, 2020 at 10:29 AM wrote: > > > > > Thanks Craig, > > > > > > Why is restore inherently that much more complicated than dump? > > > It seems like config.pl already has a number of parameters built-in > > > for both including rsync args and pre/post restore commands. > > > > > > Conceptually, I would think that what one needs to specify each time > > > is: > > > 1. Host > > > 2. Backup number > > > 3. Share > > > 4. Additional includes/excludes to determine what to restore > > > 5. Option to "delete" files no longer found > > > 6. Path to root of restore. > > > > > > Otherwise, existing includes/excludes would be assumed... > > > > > > I'm sure one could make it more complicated but am I missing something > > > basic??? > > > > > > The reality is that rsync + backuppc is really awesome... and I can do > > > (and automate) so much more with CLI and scripts than with a CGI. > > > > > > Jeff > > > > > > Craig Barratt via BackupPC-users wrote at about 09:32:48 -0700 on > > > Wednesday, May 20, 2020: > > > > Jeff, > > > > > > > > BackupPC_restore takes quite a few parameters, so many years ago I > > > decided > > > > to pass those parameters via a file rather than command-line > arguments. > > > > Probably a bad choice... > > > > > > > > There are two alternatives: > > > > > > > >- write a script that creates the restore request parameter > file (see > > > >lib/BackupPC/CGI/Restore.pm) and then runs BackupPC_restore > > > >- directly run rsync_bpc using an example of the many arguments > from > > > a > > > >successful restore log file > > > > > > > > The drawback of the 2nd approach is that information about the > restore > > > > isn't saved or logged, and you have to make sure it doesn't run > > > > concurrently with a backup. > > > > > > > > Craig > > > > > > > > > > > > On Wed, May 20, 2020 at 6:31 AM wrote: > > > > > > > > > > > > > > Is it possible to do an rsync restore from the command line using > > > > > BackupPC_restore? > > > > > If so, it's not clear from the (limited) documentation how to use > > > > > it. For example, how do you specify the desired backup number, > share, > > > > > paths, etc. > > > > > > > > > > Alternatively, is there an rsync analog of BackupPC_tarCreate? > > > > > > > > > > Said another way, rather than first creating a tar file via > > > > > BackupPC_tarCreate and then untarring (or piping through tar), is > > > > > there a way to hook directly into rsync to restore from the > command > > > line? > > > > > > > > > > This would have several advantages: > > > > > 1. It would automatically incorporate the ssh and compression > features > > > > >to restore seamlessly, efficiently, and securely across > platforms > > > > > > > > > > 2. It would allow for restoring special file types that tar > doesn't > > > > >support > > > > > > > > > > 3. It would be able to better and more exactly mirror the > parameters > > > > >given to Rsync dump (for example the same format of > 'includes' and > > > > >'excludes' > > > > > > > > > > > > > > > ___ > > > > > BackupPC-users mailing list > > > > > BackupPC-users@lists.sourceforge.net > > > > > List: > https://lists.sourceforge.net/lists/listinfo/backuppc-users > > > > > Wiki:http://backuppc.wiki.sourceforge.net > > > > > Project: http://backuppc.sourceforge.net/ > > > > > > > > > ___ > > > > BackupPC-users mailing list > > > > BackupPC-users@lists.sourceforge.net > > > > List: > https://lists.sourceforge.net/lists/listinfo/backuppc-users > > > > Wiki:http://backuppc.wiki.sourc
[BackupPC-users] how to install SCGI , exactly??
Installed... General Server Information * The servers PID is 1302, on host [redacted], version 4.3.2, started at 2020-05-21 09:18. But cant launch a backup yet - I'm stuck with the error 2020-05-21 16:44:49 scgi : BackupPC_Admin_SCGI: can't load perl SCGI module - install via CPAN; exiting in 60 seconds 2020-05-21 16:44:49 Running BackupPC_Admin_SCGI (pid=5740) 2020-05-21 16:45:49 scgi : BackupPC_Admin_SCGI: can't load perl SCGI module - install via CPAN; exiting in 60 seconds 2020-05-21 16:45:49 Running BackupPC_Admin_SCGI (pid=5744) 2020-05-21 16:46:50 scgi : BackupPC_Admin_SCGI: can't load perl SCGI module - install via CPAN; exiting in 60 seconds 2020-05-21 16:46:50 Running BackupPC_Admin_SCGI (pid=5745) 2020-05-21 16:47:50 scgi : BackupPC_Admin_SCGI: can't load perl SCGI module - install via CPAN; exiting in 60 seconds 2020-05-21 16:47:50 Running BackupPC_Admin_SCGI (pid=5746) 2020-05-21 16:48:50 scgi : BackupPC_Admin_SCGI: can't load perl SCGI module - install via CPAN; exiting in 60 seconds 2020-05-21 16:48:50 Running BackupPC_Admin_SCGI (pid=5749) 2020-05-21 16:49:50 scgi : BackupPC_Admin_SCGI: can't load perl SCGI module - install via CPAN; exiting in 60 seconds 2020-05-21 16:49:50 Running BackupPC_Admin_SCGI (pid=5751) etc etc Been looking for a workaround... I got this far: https://sourceforge.net/p/backuppc/mailman/message/36059887/ where it is suggested "You are missing the CPAN scgi module, so you should install it (eg, via CPAN). " I cannot find any details on how to do this... cpanm is installed and seems to be working.. but WHAT must I ask it to get, precisely?? # cpanm scgi ! Finding scgi on cpanmetadb failed. ! Finding scgi () on mirror http://www.cpan.org failed. ! Couldn't find module or a distribution scgi # cpanm Perl:SCGI ! Finding Perl:SCGI on cpanmetadb failed. ! Finding Perl:SCGI () on mirror http://www.cpan.org failed. ! Couldn't find module or a distribution Perl:SCGI /# cpanm Module::SCGI ! Finding Module::SCGI on cpanmetadb failed. ! Finding Module::SCGI () on mirror http://www.cpan.org failed. ! Couldn't find module or a distribution Module::SCGI etc etc... ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] replication of data pool
Hi there, On Thu, 21 May 2020, Mike Hughes wrote: we're currently syncing our cpool to an off-site location on a weekly basis. Would it be feasible to only sync the latest of each backup rather than the entire pool? To elaborate, on Saturdays we run an rsync of the entire cpool to another server to provide disaster recovery options. Is it possible/reasonable to just copy the data from the night before? Or, with de-duplication and compression, would we really save much space/transfer time? If so, what is the best way to grab just one night's worth of backups while still preserving a full recovery? Why not simply run a second BackupPC instance on the off-site server? -- 73, Ged. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
