Re: [BackupPC-users] Exclude list being ignored?
On 10 October 2012 23:10, Ian P. Christian poo...@pookey.co.uk wrote: Running: /usr/bin/ssh -q -x -l root my.hostname.com /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 --ignore-times . / snip Sent exclude: /proc Sent exclude: /sys I've just noticed on this output there is a 'Sent exclude' - I was expecting to see this in the rsync arguments. Perhaps it is working after all! How does this exclude get processed if it's not in the rsync command args? Thanks -- Blog: http://pookey.co.uk/blog Follow me on twitter: http://twitter.com/ipchristian -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
On Thu, Oct 11, 2012 at 4:30 AM, Ian P. Christian poo...@pookey.co.uk wrote: Running: /usr/bin/ssh -q -x -l root my.hostname.com /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 --ignore-times . / snip Sent exclude: /proc Sent exclude: /sys I've just noticed on this output there is a 'Sent exclude' - I was expecting to see this in the rsync arguments. Perhaps it is working after all! How does this exclude get processed if it's not in the rsync command args? That looked normal to me - are they not being excluded in the backup results? -- Les Mikesell lesmikes...@gmail.com -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
mark k wrote:
Here is how I have mine setup, note the /' = at the top, backing
up 30 + systems this way with nor errors.
$Conf{BackupFilesExclude} = {
'/' = [
'/proc/*',
'/sys/*',
'/var/run/*',
'/dev/*',
If you use '/' as the key, then these excludes only apply to machines
where you backup / (which may be all of your machines). Using '*'
means: use these excludes for all machines that don't have any
explicit excludes attached to them, which would have the same effect
if you backup / on all your machines.
We use the following with rsync over SSH and it works just fine:
$Conf{BackupFilesExclude} = {
'*' = [
'/proc',
'/var/named/run-root/proc',
'/sys',
'/mnt'
]
};
Nils Breunese.
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Here is how I have mine setup, note the /' = at the top, backing up 30 +
systems this way with nor errors.
$Conf{BackupFilesExclude} = {
'/' = [
'/proc/*',
'/sys/*',
'/var/run/*',
'/dev/*',
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Craig Barratt wrote:
Steven writes:
Running: /usr/bin/ssh -q -x -l netbackup freedom.rapidxdev.com
/usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms
--owner --group --links --times --block-size=2048 --recursive -D
--bwlimit=200 --ignore-times . /
Yes, you can see none of the excludes make it into the command.
But you knew that.
The most likely explanation is there is a per-client config.pl that
is overriding the setting. Try editing $Conf{BackupFilesExclude}
and $Conf{RsyncClientCmd} in the per-client config.pl (make
$Conf{RsyncClientCmd} something different to the main config
file to make sure the file is correctly being parsed). Then
check the actual command being run.
Craig
Yes, I know that seems like the likely answer, but unfortunately it
isn't. This happens on all of the hosts, regardless of whether or not
there are per host configuration files.
As an update, it works fine if I update the RsyncArgs directly, like so:
$Conf{RsyncArgs} = [
#
# Do not edit these!
#
'--numeric-ids',
'--perms',
'--owner',
'--group',
'--links',
'--times',
'--block-size=2048',
'--recursive',
#
# If you are using a patched client rsync that supports the
# --checksum-seed option (see http://backuppc.sourceforge.net),
# then uncomment this to enabled rsync checksum cachcing
#
#'--checksum-seed=32761',
#
# Add additional arguments here
#
'-D',
'--bwlimit=200',
'--exclude', '/proc',
'--exclude', '/sys',
'--exclude', '/tmp',
'--exclude', '/var/tmp',
'--exclude', '/usr/tmp',
'--exclude', '/mnt',
'--exclude', '/media',
'--exclude', '/auto',
'--exclude', '/var/run/acpid.socket',
'--exclude', '/var/run/dbus/system_bus_socket',
'--exclude', '/var/lib/backuppc/cpool',
'--exclude', '/var/lib/backuppc/pc',
'--exclude', '/var/lib/backuppc/pool',
'--exclude', '/var/lib/backuppc/trash',
'--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmdk',
'--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmem',
'--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmsd',
'--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmsn',
'--exclude', '/var/lib/vmware/Virtual Machines/*/*.vmss',
'--exclude', '/backupdata',
];
--
Puryear Information Technology, LLC
Baton Rouge, LA * 225-706-8414
http://www.puryear-it.com
Visit http://www.puryear-it.com/pubs/ebooks/ to download your free
copies of:
Best Practices for Managing Linux and UNIX Servers
Spam Fighting and Email Security in the 21st Century
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
Steven writes:
Yes, I know that seems like the likely answer, but unfortunately it
isn't. This happens on all of the hosts, regardless of whether or not
there are per host configuration files.
Well, if you are confident the likely explanation is wrong, I'd
recommend adding some debug statements to the code to figure
out why the value isn't being read.
Assuming you are running 3.x, in the ConfigDataRead function in
lib/BackupPC/Storage/Text.pm add the two print statements:
sub ConfigDataRead
{
my($s, $host) = @_;
my($ret, $mesg, $config, @configs);
#
# TODO: add lock
#
my $conf = {};
my $configPath = $s-ConfigPath($host);
push(@configs, $configPath) if ( -f $configPath );
foreach $config ( @configs ) {
%Conf = ();
if ( !defined($ret = do $config) ($! || $@) ) {
$mesg = Couldn't open $config: $! if ( $! );
$mesg = Couldn't execute $config: $@ if ( $@ );
$mesg =~ s/[\n\r]+//;
return ($mesg, $conf);
}
%$conf = ( %$conf, %Conf );
print(STDERR Read config file $config;\nBackupFilesExclude is
now:); # add
print(STDERR Dumper($conf-{BackupFilesExclude}));
# add
}
Then run
su - backuppc
BackupPC_dump -f -v HOSTNAME
and you should see it read each config file and the resulting BackupFilesExclude
value. Hit ^C after that.
Craig
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
[BackupPC-users] Exclude list being ignored?
We are running backuppc 2.1.2 on a debian 4 host, backing up other
debian hosts. Rsync version is 2.6.9 protocol version 29 on server and
clients.
I've set up backuppc on other systems and never run into this issue, but
whenever we run backups the $Conf{BackupFilesExclude} setting is
completely ignored and everything is backed up. Here's out backuppc config:
[EMAIL PROTECTED]:~$ cat /tmp//backuppc.config.nocomments
$ENV{'PATH'} = '/bin:/usr/bin';
delete @ENV{'IFS', 'CDPATH', 'ENV', 'BASH_ENV'};
$Conf{ServerHost} = `hostname`;
chomp($Conf{ServerHost});
$Conf{ServerPort} = -1;
$Conf{ServerMesgSecret} = '';
$Conf{MyPath} = '/bin';
$Conf{UmaskMode} = 027;
$Conf{WakeupSchedule} = [1,2,3,4,5,6,7];
$Conf{MaxBackups} = 2;
$Conf{MaxUserBackups} = 4;
$Conf{MaxPendingCmds} = 10;
$Conf{MaxBackupPCNightlyJobs} = 2;
$Conf{BackupPCNightlyPeriod} = 1;
$Conf{MaxOldLogFiles} = 14;
$Conf{DfPath} = '/bin/df';
$Conf{DfCmd} = '$dfPath $topDir';
$Conf{SplitPath} = '/usr/bin/split';
$Conf{ParPath} = '/usr/bin/par2' if -x '/usr/bin/par2';
$Conf{CatPath} = '/bin/cat';
$Conf{GzipPath} = '/bin/gzip';
$Conf{Bzip2Path} = '/usr/bin/bzip2';
$Conf{DfMaxUsagePct} = 95;
$Conf{TrashCleanSleepSec} = 300;
$Conf{DHCPAddressRanges} = [];
$Conf{BackupPCUser} = 'backuppc';
$Conf{CgiDir} = '/usr/share/backuppc/cgi-bin';
$Conf{InstallDir} = '/usr/share/backuppc';
$Conf{BackupPCUserVerify} = 1;
$Conf{HardLinkMax} = 31999;
$Conf{SmbShareName} = 'C$';
$Conf{SmbShareUserName} = '';
$Conf{SmbSharePasswd} = '';
$Conf{TarShareName} = '/';
$Conf{FullPeriod} = -1;
$Conf{IncrPeriod} = 0.97;
$Conf{FullKeepCnt} = 1;
$Conf{FullKeepCntMin} = 1;
$Conf{FullAgeMax} = 90;
$Conf{IncrKeepCnt} = 6;
$Conf{IncrKeepCntMin} = 1;
$Conf{IncrAgeMax} = 30;
$Conf{PartialAgeMax} = 3;
$Conf{IncrFill} = 0;
$Conf{RestoreInfoKeepCnt} = 10;
$Conf{ArchiveInfoKeepCnt} = 10;
$Conf{BackupFilesOnly} = undef;
$Conf{BackupFilesExclude} = {
'*' = [
'/proc',
'/sys',
'/tmp',
'/var/tmp',
'/usr/tmp',
'/mnt',
'/media',
'/auto',
'/var/run/acpid.socket',
'/var/run/dbus/system_bus_socket',
'/var/lib/backuppc/cpool',
'/var/lib/backuppc/pc',
'/var/lib/backuppc/pool',
'/var/lib/backuppc/trash',
'/var/lib/vmware/Virtual Machines/*/*.vmdk',
'/var/lib/vmware/Virtual Machines/*/*.vmem',
'/var/lib/vmware/Virtual Machines/*/*.vmsd',
'/var/lib/vmware/Virtual Machines/*/*.vmsn',
'/var/lib/vmware/Virtual Machines/*/*.vmss',
'/backupdata'
]
};
$Conf{BlackoutBadPingLimit} = 3;
$Conf{BlackoutGoodCnt} = 7;
$Conf{BlackoutPeriods} = [
{
hourBegin = 7.0,
hourEnd = 19.5,
weekDays = [1, 2, 3, 4, 5],
},
];
$Conf{BackupZeroFilesIsFatal} = 1;
$Conf{XferMethod} = 'rsync';
$Conf{XferLogLevel} = 1;
$Conf{SmbClientPath} = '/usr/bin/smbclient';
$Conf{SmbClientFullCmd} = '$smbClientPath $host\\$shareName'
. ' $I_option -U $userName -E -N -d 1'
. ' -c tarmode\\ full -Tc$X_option - $fileList';
$Conf{SmbClientIncrCmd} = '$smbClientPath $host\\$shareName'
. ' $I_option -U $userName -E -N -d 1'
. ' -c tarmode\\ full -TcN$X_option $timeStampFile - $fileList';
$Conf{SmbClientRestoreCmd} = '$smbClientPath $host\\$shareName'
. ' $I_option -U $userName -E -N -d 1'
. ' -c tarmode\\ full -Tx -';
$Conf{TarClientCmd} = '$sshPath -q -x -n -l root $host'
. ' /usr/bin/env LC_ALL=C $tarPath -c -v -f - -C
$shareName+'
. ' --totals';
$Conf{TarFullArgs} = '$fileList+';
$Conf{TarIncrArgs} = '--newer=$incrDate+ $fileList+';
$Conf{TarClientRestoreCmd} = '$sshPath -q -x -l root $host'
. ' /usr/bin/env LC_ALL=C $tarPath -x -p
--numeric-owner --same-owner'
. ' -v -f - -C $shareName+';
$Conf{TarClientPath} = '/bin/tar';
$Conf{RsyncClientPath} = '/usr/bin/rsync';
$Conf{RsyncClientCmd} = '$sshPath -q -x -l netbackup $host sudo
$rsyncPath $argList+';
$Conf{RsyncClientRestoreCmd} = '$sshPath -q -x -l netbackup $host sudo
$rsyncPath $argList+';
$Conf{RsyncShareName} = '/';
$Conf{RsyncdClientPort} = 873;
$Conf{RsyncdUserName} = '';
$Conf{RsyncdPasswd} = '';
$Conf{RsyncdAuthRequired} = 1;
$Conf{RsyncCsumCacheVerifyProb} = 0.01;
$Conf{RsyncArgs} = [
'--numeric-ids',
'--perms',
'--owner',
'--group',
'--links',
'--times',
'--block-size=2048',
'--recursive',
'-D',
'--bwlimit=200',
];
$Conf{RsyncRestoreArgs} = [
'--numeric-ids',
'--perms',
'--owner',
'--group',
'--devices',
'--links',
'--times',
'--block-size=2048',
'--relative',
'--ignore-times',
'--recursive',
];
$Conf{ArchiveDest} = '/tmp';
$Conf{ArchiveComp} = 'gzip';
$Conf{ArchivePar} = 0;
$Conf{ArchiveSplit} = 0;
$Conf{ArchiveClientCmd} =
Re: [BackupPC-users] Exclude list being ignored?
Steven writes:
We are running backuppc 2.1.2 on a debian 4 host, backing up other
debian hosts. Rsync version is 2.6.9 protocol version 29 on server and
clients.
I've set up backuppc on other systems and never run into this issue, but
whenever we run backups the $Conf{BackupFilesExclude} setting is
completely ignored and everything is backed up. Here's out backuppc config:
I'm a little suspicious of whether the sudo will cause a problem
with shell-meta characters - I'm not sure if an extra shell is
run by ssh in this case.
Try this:
- Try removing the '/var/lib/vmware/Virtual Machines...' excludes (the
ones with meta characters)
- Try replacing sudo with a full path:
$Conf{RsyncClientCmd} = '$sshPath -q -x -l netbackup $host sudo...';
In each case, you can run the backup manually with the -v flag to
see what rsync command is run, and then hit ^C to abort after you
see whether the excludes are working or not. Please post the rsync
command and whether it worked or not.
Craig
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Exclude list being ignored?
I changed the exclude list to:
$Conf{BackupFilesExclude} = {
'*' = [
'/proc',
'/sys',
'/tmp',
'/var/tmp',
'/usr/tmp',
'/mnt',
'/media',
'/auto',
'/var/run/acpid.socket',
'/var/run/dbus/system_bus_socket',
'/var/lib/backuppc/cpool',
'/var/lib/backuppc/pc',
'/var/lib/backuppc/pool',
'/var/lib/backuppc/trash',
#'/var/lib/vmware/Virtual Machines/*/*.vmdk',
#'/var/lib/vmware/Virtual Machines/*/*.vmem',
#'/var/lib/vmware/Virtual Machines/*/*.vmsd',
#'/var/lib/vmware/Virtual Machines/*/*.vmsn',
#'/var/lib/vmware/Virtual Machines/*/*.vmss',
'/backupdata'
]
};
And the rsync command to:
$Conf{RsyncClientCmd} = '$sshPath -q -x -l netbackup $host /usr/bin/sudo
$rsyncPath $argList+';
and ran the backup again. It did not work, sadly. Output:
-
Running: /usr/bin/ssh -q -x -l netbackup freedom.rapidxdev.com
/usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms
--owner --group --links --times --block-size=2048 --recursive -D
--bwlimit=200 --ignore-times . /
Xfer PIDs are now 13052
xferPids 13052
Got remote protocol 29
Negotiated protocol version 26
Remote[2]: file has vanished: /proc/2/exe
Remote[2]: file has vanished: /proc/2/task/2/exe
Remote[2]: file has vanished: /proc/3/exe
Remote[2]: file has vanished: /proc/3/task/3/exe
Remote[2]: file has vanished: /proc/4/exe
Remote[2]: file has vanished: /proc/4/task/4/exe
Remote[2]: file has vanished: /proc/5/exe
Remote[2]: file has vanished: /proc/5/task/5/exe
Remote[2]: file has vanished: /proc/6/exe
Remote[2]: file has vanished: /proc/6/task/6/exe
Remote[2]: file has vanished: /proc/7/exe
Remote[2]: file has vanished: /proc/7/task/7/exe
Remote[2]: file has vanished: /proc/11/exe
Remote[2]: file has vanished: /proc/11/task/11/exe
Remote[2]: file has vanished: /proc/14/exe
Remote[2]: file has vanished: /proc/14/task/14/exe
Remote[2]: file has vanished: /proc/16/exe
Remote[2]: file has vanished: /proc/16/task/16/exe
Remote[2]: file has vanished: /proc/92/exe
Remote[2]: file has vanished: /proc/92/task/92/exe
Remote[2]: file has vanished: /proc/93/exe
Remote[2]: file has vanished: /proc/93/task/93/exe
Remote[2]: file has vanished: /proc/94/exe
Remote[2]: file has vanished: /proc/94/task/94/exe
Remote[2]: file has vanished: /proc/95/exe
Remote[2]: file has vanished: /proc/95/task/95/exe
Remote[2]: file has vanished: /proc/601/exe
Remote[2]: file has vanished: /proc/601/task/601/exe
Remote[2]: file has vanished: /proc/602/exe
Remote[2]: file has vanished: /proc/602/task/602/exe
Remote[2]: file has vanished: /proc/746/exe
Remote[2]: file has vanished: /proc/746/task/746/exe
Remote[2]: file has vanished: /proc/747/exe
Remote[2]: file has vanished: /proc/747/task/747/exe
Remote[2]: file has vanished: /proc/748/exe
Remote[2]: file has vanished: /proc/748/task/748/exe
Remote[2]: file has vanished: /proc/749/exe
Remote[2]: file has vanished: /proc/749/task/749/exe
Remote[2]: file has vanished: /proc/750/exe
Remote[2]: file has vanished: /proc/750/task/750/exe
Remote[2]: file has vanished: /proc/751/exe
Remote[2]: file has vanished: /proc/751/task/751/exe
Remote[2]: file has vanished: /proc/995/exe
Remote[2]: file has vanished: /proc/995/task/995/exe
Remote[2]: file has vanished: /proc/999/exe
Remote[2]: file has vanished: /proc/999/task/999/exe
Remote[2]: file has vanished: /proc/1003/exe
Remote[2]: file has vanished: /proc/1003/task/1003/exe
Remote[2]: file has vanished: /proc/1032/exe
Remote[2]: file has vanished: /proc/1032/task/1032/exe
Remote[2]: file has vanished: /proc/1092/exe
Remote[2]: file has vanished: /proc/1092/task/1092/exe
Remote[2]: file has vanished: /proc/1673/exe
Remote[2]: file has vanished: /proc/1673/task/1673/exe
Remote[2]: file has vanished: /proc/1934/exe
Remote[2]: file has vanished: /proc/1934/task/1934/exe
Remote[2]: file has vanished: /proc/2819/exe
Remote[2]: file has vanished: /proc/2819/task/2819/exe
Xfer PIDs are now 13052,13055
xferPids 13052,13055
create d 755 0/04096 .
create d 755 0/04096 .vmware
create d 755 0/04096 backupdata
same 644 0/01853 backupdata/backups
create d 755 0/04096 backupdata/cache
create d 755 0/04096 backupdata/cache/apt
create d 755 0/0 24576 backupdata/cache/apt/archives
same 444 0/0 179206
backupdata/cache/apt/archives/adduser_3.102
_all.deb
create 444 0/0 197976
backupdata/cache/apt/archives/apt-utils_0.6
.46.4-0.1_amd64.deb
Parent read EOF from child: fatal error!
Done: 0 files, 0 bytes
Got fatal error during xfer (Child exited prematurely)
Backup aborted by user signal
dump failed: Child exited prematurely
link
Re: [BackupPC-users] Exclude list being ignored?
Steven writes:
Running: /usr/bin/ssh -q -x -l netbackup freedom.rapidxdev.com
/usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms
--owner --group --links --times --block-size=2048 --recursive -D
--bwlimit=200 --ignore-times . /
Yes, you can see none of the excludes make it into the command.
But you knew that.
The most likely explanation is there is a per-client config.pl that
is overriding the setting. Try editing $Conf{BackupFilesExclude}
and $Conf{RsyncClientCmd} in the per-client config.pl (make
$Conf{RsyncClientCmd} something different to the main config
file to make sure the file is correctly being parsed). Then
check the actual command being run.
Craig
-
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse012070mrt/direct/01/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
