Hi,
David Mansfield wrote on 20.09.2007 at 09:55:28 [[BackupPC-users] is it
possible to verify the cpool?]:
> I have recently had to transfer the entire backuppc installation to a
> new server due to the demise of the old one. I am transferring the
> cpool/ and pc/ directories using tar, which seems to be preserving the
> hard links (in doing so it's chewing up VAST amounts of memory on the
> 'create' side, something like 2.5gb of ram for a 250GB pool!). Once I
> get it, I'd like to verify the MD5 sums of the cpool to make sure that
> the files got transferred ok. In fact, I expect there to be some
> problems because the disk began to get some I/O errors.
>
> Is there a BackupPC_verify perl script anywhere?
sorry for the delay in answering - I don't find much time for keeping up
with the list lately.
Actually, I wrote a script which attempts to do just that a few months ago.
You could have found it in the list archives, I believe. I've attached it in
case it is still of use to you.
As for your question regarding linking to corrupt pool files, that should be
no problem. BackupPC needs to read the entire pool file and compare it with
the incoming data to determine if it matches, as different contents can have
the same checksum. A corrupt *compressed* pool file is highly unlikely to
match (and if it did, that would be fine, because it would actually contain
the contents you are now trying to store :). So: BackupPC will only link
newly transfered files to pool files with exactly matching contents.
Note that, as Carl said, incremental backups will *not* correct errors in
damaged files that have not changed since the reference backup. But you will
probably have restarted with a full backup anyway :).
Regards,
Holger
#!/usr/bin/perl
#============================================================= -*-perl-*-
#
# BackupPC_verifyPool: Verify pool integrity
#
# DESCRIPTION
#
# BackupPC_verifyPool tries to verify the integrity of the pool files,
# based on their file names, which are supposed to correspond to MD5
# digests calculated from the (uncompressed) length and parts of their
# (again uncompressed) contents.
# Needs to be run as backuppc user for access to the pool files and
# meta data.
#
# Usage: BackupPC_verifyPool [-v] [-p] [-u] [-r range] [-s]
#
# Options:
#
# -v Show what is going on. Without this flag, only errors found
# are displayed, which might be very boring. Use '-p' for a
# bit of entertainment without causing your tty to scroll so
# much.
# -p Show more terse progress output.
# -u Check the pool (uncompressed files), not the cpool
# (uncompressed files).
# -r range Specify the pool range to check (see below). 'range'
# can be a Perl expression like '0 .. 255' or '0, 10, 30 .. 40'.
# Only safe characters allowed [\da-fA-F,.\s].
# -s Show summary.
#
# The pool range was chosen as in BackupPC_nightly as means to divide
# the possibly lengthy operation of verifying the pool into smaller
# steps. The full pool goes from 0 to 255, that's all you really need to
# know. For more information, see BackupPC_nightly from the BackupPC
# distribution.
#
# AUTHOR
# Holger Parplies <wopp at parplies.de>
#
# VERSION
# $Id$
#
# COPYRIGHT
# Copyright (C) 2007 Holger Parplies
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
#========================================================================
use strict;
use lib '/usr/share/backuppc/lib'; # Debian; change to fit your needs
use BackupPC::Lib;
use BackupPC::FileZIO;
use Getopt::Std;
use File::Find;
# $ENV {PATH} = '/bin:/usr/bin';
# chdir '.';
my %opts = (
v => 0, # verbose output
p => 0, # terse progress output
u => 0, # pool (1) or cpool (0)
r => '0 .. 255', # range
s => 0, # show summary
);
unless (getopts ('vpur:s', \%opts)) {
die <<EOM;
Usage: $0 [-v] [-p] [-u] [-r range] [-s]
Options: -v Verbose output of files being checked and result
-p Terse output of what is happening (counter)
-u Check uncompressed pool (pool instead of cpool)
-r range Check subset of pool (default whole pool = 0 .. 255).
Use any valid Perl expression containing only numbers
(octal or hexadecimal is ok), dots, commas and whitespace
here.
-s Show summary.
EOM
}
# unbuffered output
$| = 1;
# some variables
my $bpc = new BackupPC::Lib # BackupPC object
or die "Can't create BackupPC object!\n";
my $pooldir = $bpc -> {TopDir} . '/' . ($opts {u} ? 'pool' : 'cpool') . '/';
# Pool directory to check
my @range; # Pool range to check
my $mismatch_count = 0; # number of invalid files
my $file_count = 0; # running file count for progress output
my $md5; # handle of MD5 object
# # untaint range specification; NOTE THAT THIS IS NOT SECURE!
# $opts {r} = $1
# if $opts {r} =~ /^(.*)$/;
# check range specification
die "Range specification '$opts{r}' contains insecure characters!\n"
if $opts {r} !~ /^[0-9a-fA-Fx.,\s]*$/;
eval "[EMAIL PROTECTED] = ($opts{r});";
if ($@) {
die "Range specification '$opts{r}' is invalid: [EMAIL PROTECTED]";
} elsif (not defined @range or @range == 0) {
die "Range specification '$opts{r}' is empty. Nothing to do.\n";
} elsif (grep { $_ < 0 or $_ > 255 } @range) {
die "Range specification '$opts{r}' contains values outside (0 .. 255)\n";
}
# iterate over the specified part of the pool
$md5 = new Digest::MD5
or die "Can't create MD5 object: $!\n";
foreach my $i (@range) {
my $dir = sprintf '%s/%1x/%1x', $pooldir, int ($i / 16), $i % 16;
find (\&validate_pool_dir, $dir)
if -d $dir;
}
# Summary
if ($opts {s}) {
printf "%d files in %d directories checked, %d had wrong digests.\n",
$file_count, @range * 16, $mismatch_count;
} elsif ($mismatch_count > 0) {
print "ERROR: $mismatch_count files in ", $opts {u} ? 'pool' : 'cpool',
", range ($opts{r}), seem to be corrupt!\n";
}
# Return code
exit $mismatch_count > 0 ? 1 : 0;
# actual verification process
sub validate_pool_dir {
my ($name_md5) = ($_ =~ /^([0-9a-fA-F]{32})/);
my $content_md5;
return # ignore directories
if -d $File::Find::name;
$file_count ++;
if ($opts {p} and not $opts {v}) {
if (/^([0-9a-fA-F])([0-9a-fA-F])/) {
print "[$1$2 $file_count]\r";
} else {
print "[ $file_count]\r";
}
}
# since we are only reading the file, we can treat the 'compLevel' parameter
# of BackupPC::FileZIO::open as a boolean
my $fh = BackupPC::FileZIO -> open ($File::Find::name, 0, ! $opts {u});
if (defined $fh) {
my $buf;
my $bytes = $fh -> read (\$buf, 1024 * 1024 + 100);
if ($bytes > 1024 * 1024) {
# read complete file for determining length. Keep first 1MB in $buf,
# put total length into $bytes, read in 100KB chunks for lower mem usage
my $buf2;
my $new = 1;
while ($new > 0) {
$new = $fh -> read (\$buf2, 102400);
$bytes += $new;
}
}
$content_md5 = $bpc -> Buffer2MD5 ($md5, $bytes, \$buf);
if ($content_md5 ne $name_md5) {
# print ">$content_md5< != >$name_md5<\n";
printf "[%5d] %-36.36s != %-32.32s\n", $file_count, $_, $content_md5;
} elsif ($opts {v}) {
printf "[%5d] %-36.36s (%10d) ok\n", $file_count, $_, $bytes;
}
} else {
# open failed, count as mismatch
print "$_: BackupPC::FileZIO::open failed!\n";
$mismatch_count ++;
}
}
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
