[BackupPC-users] BackupFilesExcludes using rsync being ignored
Hi all this appears to be a common question for new backuppc users. Despite trawling this mailing list and google I am still unable to find a solution. A newly built (due disk failure) Ubuntu 14.04 server running Backuppc 3.2.1 from Ubuntu repositories. Backing up a load of Linux servers with RsyncShareName http://1.1.2.3/backuppc/index.cgi?action=viewtype=docs#_conf_rsyncsharename_ set as / I want to then exclude subdirectories eg /proc, /sys, /tmp etc. I've copied the setup from another working backup server on which the excludes have all been added via the GUI with the paths as new keys so the /etc/backuppc/config.pl excludes section looks like this: $Conf{BackupFilesExclude} = { '/var/tmp' = [ '' ], '/pagefile.sys' = [ '' ], '/var/spool' = [ '' ], The old server works as expected but on the new server the excluded list is being ignored and so the pool is filling up with contents of /proc etc. Most forum posts I've read show a different syntax in the /etc/backuppc/config.pl file which I've managed to emulate by adding the exclusions to the GUI by adding / in the key then using the insert button to add excluded directories. This ends up with config.pl file looking more like what others are posting here but the exclusion files are still not being read. $Conf{BackupFilesExclude} = { '/' = [ 'proc', 'sys', '*/Cache', 'RECYCLER', 'System Volume Information', 'dev', 'lost+found', 'tmp', I am fairly sure I have the syntax of the exclude list correct now, though advice to the contrary is welcome. I've also tried editing the config.pl file using the * wildcard and am trying to run backup now but I suspect this will fail too. $Conf{BackupFilesExclude} = { '*' = [ 'proc', 'sys', Am I missing something rudimentary here? I've compared a working and non-working backuppc server and cannot see anything different in the 2 configs, at least til I rejigged things above so am at a loss as to why one works and the other does not. Any pointers gratefully received. regards -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes using rsync being ignored
Tom Fallon wrote: $Conf{BackupFilesExclude} = { '*' = [ 'proc', 'sys', My list shows: $Conf{BackupFilesExclude} = { '*' = [ '*/Cache', '*.mp3', '*.MP3', '*.m4a', '*.M4A', '*.m4v', ] }; Doug -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes using rsync being ignored
hi, This works for me on my Ubuntu 14.04 backuppc server: $Conf{BackupFilesExclude} = { '*' = [ '/mnt', '/net', '/tmp', '/proc', '/sys', '/dev', '/run' ] }; Mark On 10/12/2014 4:27 PM, Tom Fallon wrote: Hi all this appears to be a common question for new backuppc users. Despite trawling this mailing list and google I am still unable to find a solution. A newly built (due disk failure) Ubuntu 14.04 server running Backuppc 3.2.1 from Ubuntu repositories. Backing up a load of Linux servers with *MailScanner has detected a possible fraud attempt from 1.1.2.3 claiming to be* RsyncShareName http://1.1.2.3/backuppc/index.cgi?action=viewtype=docs#_conf_rsyncsharename_ set as / I want to then exclude subdirectories eg /proc, /sys, /tmp etc. I've copied the setup from another working backup server on which the excludes have all been added via the GUI with the paths as new keys so the /etc/backuppc/config.pl excludes section looks like this: $Conf{BackupFilesExclude} = { '/var/tmp' = [ '' ], '/pagefile.sys' = [ '' ], '/var/spool' = [ '' ], The old server works as expected but on the new server the excluded list is being ignored and so the pool is filling up with contents of /proc etc. Most forum posts I've read show a different syntax in the /etc/backuppc/config.pl file which I've managed to emulate by adding the exclusions to the GUI by adding / in the key then using the insert button to add excluded directories. This ends up with config.pl file looking more like what others are posting here but the exclusion files are still not being read. $Conf{BackupFilesExclude} = { '/' = [ 'proc', 'sys', '*/Cache', 'RECYCLER', 'System Volume Information', 'dev', 'lost+found', 'tmp', I am fairly sure I have the syntax of the exclude list correct now, though advice to the contrary is welcome. I've also tried editing the config.pl file using the * wildcard and am trying to run backup now but I suspect this will fail too. $Conf{BackupFilesExclude} = { '*' = [ 'proc', 'sys', Am I missing something rudimentary here? I've compared a working and non-working backuppc server and cannot see anything different in the 2 configs, at least til I rejigged things above so am at a loss as to why one works and the other does not. Any pointers gratefully received. regards -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes using rsync being ignored
Thanks Doug by my limited understanding, the * wildcard in my example below would mean the system would exclude any directory called proc or sys etc regardless of the share, whereas what I think should be used is ryncshare = / and then specify the directories under there explicitly which I had done. Amending to '*' = [ in the config.pl file resulted in same behaviour I've been seeing all along. In your config I'd expect it to be excluding file types by extension rather than directories. I could try '*/proc' etc I guess but think thats a bandaid approach and would really like to get this working the way its supposed to based on the documentation. regards Tom On 12/10/14 22:32, Doug Lytle wrote: Tom Fallon wrote: $Conf{BackupFilesExclude} = { '*' = [ 'proc', 'sys', My list shows: $Conf{BackupFilesExclude} = { '*' = [ '*/Cache', '*.mp3', '*.MP3', '*.m4a', '*.M4A', '*.m4v', ] }; Doug -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Tom Fallon 1033 Aikenhead Road Glasgow G44 4QT Tel: 0141 5697824 Mob: 07909 754 170 Skype: fallontom -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes using rsync being ignored
Thanks Mark I had tried the * wildcard which did not work but had not prepended the / before each directory. I'll give that a go next regards Tom On 12/10/14 21:45, Mark Maciolek wrote: hi, This works for me on my Ubuntu 14.04 backuppc server: $Conf{BackupFilesExclude} = { '*' = [ '/mnt', '/net', '/tmp', '/proc', '/sys', '/dev', '/run' ] }; Mark On 10/12/2014 4:27 PM, Tom Fallon wrote: Hi all this appears to be a common question for new backuppc users. Despite trawling this mailing list and google I am still unable to find a solution. A newly built (due disk failure) Ubuntu 14.04 server running Backuppc 3.2.1 from Ubuntu repositories. Backing up a load of Linux servers with RSyncNamehttp://1.1.2.3/backuppc/index.cgi?action=viewtype=docs#_conf_rsyncsharename_ set as / I want to then exclude subdirectories eg /proc, /sys, /tmp etc. I've copied the setup from another working backup server on which the excludes have all been added via the GUI with the paths as new keys so the /etc/backuppc/config.pl excludes section looks like this: $Conf{BackupFilesExclude} = { '/var/tmp' = [ '' ], '/pagefile.sys' = [ '' ], '/var/spool' = [ '' ], The old server works as expected but on the new server the excluded list is being ignored and so the pool is filling up with contents of /proc etc. Most forum posts I've read show a different syntax in the /etc/backuppc/config.pl file which I've managed to emulate by adding the exclusions to the GUI by adding / in the key then using the insert button to add excluded directories. This ends up with config.pl file looking more like what others are posting here but the exclusion files are still not being read. $Conf{BackupFilesExclude} = { '/' = [ 'proc', 'sys', '*/Cache', 'RECYCLER', 'System Volume Information', 'dev', 'lost+found', 'tmp', I am fairly sure I have the syntax of the exclude list correct now, though advice to the contrary is welcome. I've also tried editing the config.pl file using the * wildcard and am trying to run backup now but I suspect this will fail too. $Conf{BackupFilesExclude} = { '*' = [ 'proc', 'sys', Am I missing something rudimentary here? I've compared a working and non-working backuppc server and cannot see anything different in the 2 configs, at least til I rejigged things above so am at a loss as to why one works and the other does not. Any pointers gratefully received. regards -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project:http://backuppc.sourceforge.net/ -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Tom Fallon 1033 Aikenhead Road Glasgow G44 4QT Tel: 0141 5697824 Mob: 07909 754 170 Skype: fallontom -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes using rsync being ignored
On 13/10/14 08:54, Tom Fallon wrote: Thanks Doug by my limited understanding, the * wildcard in my example below would mean the system would exclude any directory called proc or sys etc regardless of the share, whereas what I think should be used is ryncshare = / and then specify the directories under there explicitly which I had done. Amending to '*' = [ in the config.pl file resulted in same behaviour I've been seeing all along. In your config I'd expect it to be excluding file types by extension rather than directories. I could try '*/proc' etc I guess but think thats a bandaid approach and would really like to get this working the way its supposed to based on the documentation. regards Tom On 12/10/14 22:32, Doug Lytle wrote: Tom Fallon wrote: $Conf{BackupFilesExclude} = { '*' = [ 'proc', 'sys', My list shows: $Conf{BackupFilesExclude} = { '*' = [ '*/Cache', '*.mp3', '*.MP3', '*.m4a', '*.M4A', '*.m4v', ] }; $Conf{BackupFilesExclude} = { '/' = [ '/proc/' '/tmp/' ] }; The '/' is the share name, so using * just means all shares, since you only have one share, then there is no difference. I would suggest using /proc/ (trailing slash) because then you will backup the directory without the contents. After a full restore, you will need the directory /proc to exist, or else mounting /proc will fail, same with /tmp etc. Also, be careful when excluding /var/log or similar, as some software will want subdirectories to exist. Also, you could consider add --one-file-system to the rsync command which will automatically skip any other mounted drives/partitions. Finally, if it is still not working, please post the full config file, along with the log from a full backup run (at least the start where it shows what command was sent, and a sample where it has backed up some files that you had requested). Regards, Adam -- Adam Goryachev Website Managers www.websitemanagers.com.au -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes using rsync being ignored
Thanks Adam that makes a lot of sense. Trying now and will confirm if this works. I do find it really odd that the same config works on one machine but not another though. regards Tom On 12/10/14 23:34, Adam Goryachev wrote: On 13/10/14 08:54, Tom Fallon wrote: Thanks Doug by my limited understanding, the * wildcard in my example below would mean the system would exclude any directory called proc or sys etc regardless of the share, whereas what I think should be used is ryncshare = / and then specify the directories under there explicitly which I had done. Amending to '*' = [ in the config.pl file resulted in same behaviour I've been seeing all along. In your config I'd expect it to be excluding file types by extension rather than directories. I could try '*/proc' etc I guess but think thats a bandaid approach and would really like to get this working the way its supposed to based on the documentation. regards Tom On 12/10/14 22:32, Doug Lytle wrote: Tom Fallon wrote: $Conf{BackupFilesExclude} = { '*' = [ 'proc', 'sys', My list shows: $Conf{BackupFilesExclude} = { '*' = [ '*/Cache', '*.mp3', '*.MP3', '*.m4a', '*.M4A', '*.m4v', ] }; $Conf{BackupFilesExclude} = { '/' = [ '/proc/' '/tmp/' ] }; The '/' is the share name, so using * just means all shares, since you only have one share, then there is no difference. I would suggest using /proc/ (trailing slash) because then you will backup the directory without the contents. After a full restore, you will need the directory /proc to exist, or else mounting /proc will fail, same with /tmp etc. Also, be careful when excluding /var/log or similar, as some software will want subdirectories to exist. Also, you could consider add --one-file-system to the rsync command which will automatically skip any other mounted drives/partitions. Finally, if it is still not working, please post the full config file, along with the log from a full backup run (at least the start where it shows what command was sent, and a sample where it has backed up some files that you had requested). Regards, Adam -- Tom Fallon 1033 Aikenhead Road Glasgow G44 4QT Tel: 0141 5697824 Mob: 07909 754 170 Skype: fallontom -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupFilesExcludes using rsync being ignored
Hi Adam I amended to using * as the key then putting each directory to exclude with a pre and post / e.g. /proc/ but still seeing the directories being excluded. Start of Xferlog error log file: Contents of file /var/lib/backuppc/pc/server.domain.com/XferLOG.0.z, modified 2014-10-13 00:58:18 (Extracting only Errors) full backup started for directory / Running: /usr/bin/ssh -q -x -l userserver.domain.com /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . / Xfer PIDs are now 973 Got remote protocol 31 Negotiated protocol version 28 Remote[1]: rsync: opendir /root failed: Permission denied (13) Remote[1]: rsync: opendir /opt/fog/utils failed: Permission denied (13) Remote[1]: rsync: opendir /opt/sophos-av failed: Permission denied (13) Remote[1]: rsync: opendir /var/lib/sudo failed: Permission denied (13) Remote[1]: rsync: opendir /var/lib/php5 failed: Permission denied (13) Remote[1]: rsync: opendir /var/lib/mysql failed: Permission denied (13) Remote[1]: rsync: opendir /var/lib/polkit-1 failed: Permission denied (13) ... ... Log file for the host only shows the following: 2014-10-13 00:04:07 full backup started for directory / 2014-10-13 00:57:26 full backup 0 complete, 324323 files, 19739494410 bytes, 51258 xferErrs (0 bad files, 0 bad shares, 51258 other) But I can navigate the backup folders and can see for instance content under the /etc folder (which should be there) but also under /proc/ which should be excluded the nightly cleanup is currently running so can't get to the config.pl file but will post in the morning regards Tom On 12/10/14 23:34, Adam Goryachev wrote: On 13/10/14 08:54, Tom Fallon wrote: Thanks Doug by my limited understanding, the * wildcard in my example below would mean the system would exclude any directory called proc or sys etc regardless of the share, whereas what I think should be used is ryncshare = / and then specify the directories under there explicitly which I had done. Amending to '*' = [ in the config.pl file resulted in same behaviour I've been seeing all along. In your config I'd expect it to be excluding file types by extension rather than directories. I could try '*/proc' etc I guess but think thats a bandaid approach and would really like to get this working the way its supposed to based on the documentation. regards Tom On 12/10/14 22:32, Doug Lytle wrote: Tom Fallon wrote: $Conf{BackupFilesExclude} = { '*' = [ 'proc', 'sys', My list shows: $Conf{BackupFilesExclude} = { '*' = [ '*/Cache', '*.mp3', '*.MP3', '*.m4a', '*.M4A', '*.m4v', ] }; $Conf{BackupFilesExclude} = { '/' = [ '/proc/' '/tmp/' ] }; The '/' is the share name, so using * just means all shares, since you only have one share, then there is no difference. I would suggest using /proc/ (trailing slash) because then you will backup the directory without the contents. After a full restore, you will need the directory /proc to exist, or else mounting /proc will fail, same with /tmp etc. Also, be careful when excluding /var/log or similar, as some software will want subdirectories to exist. Also, you could consider add --one-file-system to the rsync command which will automatically skip any other mounted drives/partitions. Finally, if it is still not working, please post the full config file, along with the log from a full backup run (at least the start where it shows what command was sent, and a sample where it has backed up some files that you had requested). Regards, Adam -- Tom Fallon 1033 Aikenhead Road Glasgow G44 4QT Tel: 0141 5697824 Mob: 07909 754 170 Skype: fallontom -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Compression Experiences?
Hi all, I remember having read about restoring single files from command line needs some BackupPC specific script or tricks to uncompress the files when using ocmpression for BackupPC. For a new instance I'm thinking of storing the files without compression to be able to easily restore them directly from command line if needed. Is there anyone out here who has some experience about the average compression ratio in BackupPC? I know it depends on the type of data- most of them are office documents (Word and OpenOffice). Just share your experience, thanks Christian -- Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://p.sf.net/sfu/Zoho ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/