Please see a rather lengthy and perhaps overly complicated discussion on
what could be a problem with the V4 alpha3.
I've recently upgraded from BackupPC v3.2.1 to BackupPC 4.0.0Alpha 3
I'm seeing a lot of messages like this:
__bpc_progress_fileCnt__ 36041
__bpc_progress_fileCnt__ 36061
G bpc_fileOpen: can't open V3 file
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postinst
(from var/lib/dpkg/info/zlib1g:i386.postinst, 9, 0)
G bpc_fileOpen: can't open V3 file
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postinst
(from var/lib/dpkg/info/zlib1g:i386.postinst, 9, 0)
rsync_bpc: failed to open "/var/lib/dpkg/info/zlib1g:i386.postinst",
continuing: No such file or directory (2)
G bpc_fileOpen: can't open V3 file
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postrm
(from var/lib/dpkg/info/zlib1g:i386.postrm, 9, 0)
G bpc_fileOpen: can't open V3 file
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postrm
(from var/lib/dpkg/info/zlib1g:i386.postrm, 9, 0)
rsync_bpc: failed to open "/var/lib/dpkg/info/zlib1g:i386.postrm",
continuing: No such file or directory (2)
__bpc_progress_fileCnt__ 36081
R bpc_fileOpen: can't open V3 file
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/fcache/fman/fda/findex.db
(from var/cache/man/da/index.db, 9, 0)
R bpc_fileOpen: can't open V3 file
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/fcache/fman/fda/f.index.db.000000
(from var/cache/man/da/.index.db.000000, 9, 0)
rsync_bpc: mkstemp "/var/cache/man/da/.index.db.000000" failed: No such
file or directory (2)
R bpc_fileOpen: can't open V3 file
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/fcache/fman/fsl/findex.db
(from var/cache/man/sl/index.db, 9, 0)
R bpc_fileOpen: can't open V3 file
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/fcache/fman/fsl/f.index.db.000000
(from var/cache/man/sl/.index.db.000000, 9, 0)
rsync_bpc: mkstemp "/var/cache/man/sl/.index.db.000000" failed: No such
file or directory (2)
__bpc_progress_fileCnt__ 36101
__bpc_progress_fileCnt__ 36121
__bpc_progress_fileCnt__ 36141
[SNIP]
R bpc_fileOpen: can't open V3 file
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postinst
(from var/lib/dpkg/info/zlib1g:i386.postinst, 9, 0)
R bpc_fileOpen: can't open V3 file
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postrm
(from var/lib/dpkg/info/zlib1g:i386.postrm, 9, 0)
R bpc_sysCall_checkFileMatch(var/lib/dpkg/triggers/Lock): couldn't
unmark pool file
/var/lib/backuppc/cpool/d4/1c/d41d8cd98f00b204e9800998ecf8427e - rewriting
__bpc_progress_fileCnt__ 36561
__bpc_progress_fileCnt__ 36581
Done: 185 errors, 98 filesExist, 96863484 sizeExist, 26854849
sizeExistComp, 0 filesTotal, 0 sizeTotal, 49 filesNew, 90478572 sizeNew,
32585824 sizeNewComp, 40055 inode
DoneGen: 362 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 36588
filesTotal, 1146156705 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp,
39680 inode
rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1542) [generator=3.0.9.3]
rsync_bpc exited with benign status 23 (5888)
Which comes from this section of the code:
/*
* need to read existing file
*/
if ( file->digest.len > 0 ) {
/*
* all non-empty V4+ files have digests, so use the digest
to look in the pool
*/
bpc_digest_md52path(fullPath, file->compress, &file->digest);
if ( bpc_fileZIO_open(&fdz, fullPath, 0, file->compress) ) {
bpc_logErrf("bpc_fileOpen: can't open pool file %s (from
%s, %d, %d)\n", fullPath, fd->fileName, file->compress, file->digest.len);
Stats.ErrorCnt++;
bpc_fileDescFree(fd);
return NULL;
}
} else {
/*
* must be a V3 file - look in the backup directory
*/
bpc_attribCache_getFullMangledPath(&acNew, fullPath,
(char*)fileName, file->backupNum);
if ( bpc_fileZIO_open(&fdz, fullPath, 0, file->compress) ) {
bpc_logErrf("bpc_fileOpen: can't open V3 file %s (from
%s, %d, %d)\n", fullPath, fd->fileName, file->compress, file->digest.len);
Stats.ErrorCnt++;
bpc_fileDescFree(fd);
return NULL;
>From my limited understanding, this says that it should be one of the
old V3 files (most likely correct), but seems to suggest it failed to
read the content of the file. Now after seeing the error, I checked, and
indeed the file does not exist:
keep:/var/lib/backuppc/pc/filesrv# ls -l
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postinst
ls: cannot access
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postinst:
No such file or directory
However, I can see the file DID exist in older backups:
keep:/var/lib/backuppc/pc/filesrv# ls -l
/var/lib/backuppc/pc/hostname/185?/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postinst
-rwxr-x--- 59299 backuppc backuppc 184 Mar 9 2009
/var/lib/backuppc/pc/hostname/1850/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postinst
-rwxr-x--- 59299 backuppc backuppc 184 Mar 9 2009
/var/lib/backuppc/pc/hostname/1853/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postinst
BTW, 1850 and 1853 were full backups from V3, 1854 is the first V4
backup being run
So, why didn't the file get copied along with all the other files before
the backup started?
There were no reported errors during the copy stage:
BackupPC_backupDuplicate: got 0 errors and 0 file open errors
So, my obvious concern is that there is some error where files are not
copied during the duplicate stage, and then during the backup stage
those files are not re-created (ie, backed up)
I can see:
/usr/share/backuppc/bin/BackupPC_attribPrint
/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/flib/fdpkg/finfo/attrib
[SNIP]
'zlib1g:i386.postinst' => {
'compress' => 9,
'digest' => '5d0e769df33e016b3c52a0971e8d258c',
'gid' => 0,
'inode' => 40049,
'mode' => 493,
'mtime' => 1340384585,
'name' => 'zlib1g:i386.postinst',
'nlinks' => 0,
'size' => 135,
'type' => 0,
'uid' => 0
},
[SNIP]
So this seems to be the problem:
ls -l /var/lib/backuppc/cpool/5/d/0/5d0e769df33e016b3c52a0971e8d258c
ls: cannot access
/var/lib/backuppc/cpool/5/d/0/5d0e769df33e016b3c52a0971e8d258c: No such
file or directory
/usr/share/backuppc/bin/BackupPC_zcat
/var/lib/backuppc/pc/hostname/1853/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postinst
#!/bin/sh
set -e
# Automatically added by dh_makeshlibs
if [ "$1" = "configure" ]; then
ldconfig
fi
# End automatically added section
So, we can see the file existed in the previous full backup, can still
be decompressed, but seems to be missing from the pool.
Here is the information from the old attrib file:
'zlib1g:i386.postinst' => {
'compress' => 9,
'digest' => '',
'gid' => 0,
'inode' => 0,
'mode' => 33261,
'mtime' => 1340384585,
'name' => 'zlib1g:i386.postinst',
'nlinks' => 0,
'size' => 135,
'type' => 0,
'uid' => 0
},
Also, uncompressing the old file, and then computing the full file md5:
/usr/share/backuppc/bin/BackupPC_zcat `pwd`/fzlib1g:i386.postinst >
/tmp/zlib1g:i386.postinst
md5sum /tmp/zlib1g\:i386.postinst
5d0e769df33e016b3c52a0971e8d258c /tmp/zlib1g:i386.postinst
The MD5 seems to match what backuppc 4 is expecting.
So, why is the pool file missing, and why doesn't this get fixed? I
suspect what will happen is that backuppc v4 would continue to report
this "error" on every future backup, but never actually fix it.
Eventually all old V3 backups will be deleted, and after that I won't be
able to recover the file (assuming this wasn't a standard OS file that I
can simply re-download and install, eg, if it was some document that I
had created/etc).
I seem to have this same issue with a number of other hosts (but not
every host has the problem, and most files are working correctly).
One thought to fix this would be to simply hardlink the file into the
pool from the old backups...
Error message:
missing=/var/lib/backuppc/pc/hostname/1854/f%2f/fvar/flib/fdpkg/finfo/fzlib1g:i386.postinst
pcdir=`echo $missing | cut -d/ -f1-6`
cd $pcdir
bkup=`echo $missing | awk -F/ '{ print $7 }'`
prevbkup=`grep full backups|grep -B1 $bkup| head -1|awk '{ print $1 }'`
file=`echo $missing | cut -d/ -f8-`
if [ -f $prevbkup/$file ]
then
#find the md5, and then link into the pool
attrib=`dirname $missing`/attrib
missingfile=`basename $missing | sed -e s/^f//`
digest=`/usr/share/backuppc/bin/BackupPC_attribPrint $attrib | egrep
"$missingfile. =>|digest" | grep -A1 $missingfile|grep digest|awk -F\'
'{print $4 }'`
# confirm it is missing from the pool both V4 and V3
v4one=`echo $digest | cut -c-2`
v4two=`echo $digest | cut -c3-4`
if [ -f /var/lib/backuppc/cpool/$v4one/$v4two/$digest ]
then
echo "ERROR: The pool file already exists... check this
manually: $missing"
else
v3one=`echo $digest | cut -c1`
v3two=`echo $digest | cut -c2`
v3three=`echo $digest | cut -c3`
if [ -f /var/lib/backuppc/cpool/$v3one/$v3two/$v3three/$digest ]
then
echo "ERROR: The pool file already exists... check this
manually: $missing"
else
# Create the needed directory
mkdir -p /var/lib/backuppc/cpool/$v4one/$v4two
# Actually link the file from the previous backup to the pool
ln $prevbkup/$file /var/lib/backuppc/cpool/$v4one/$v4two/$digest
fi
else
echo "ERROR: Couldn't fix missing file, please correct manually:
$missing"
fi
Ooops, I think I've worked out the common factor for the "missing"
files. It's the number of hardlinks. I've checked a number of them, and
they all have a large number of existing links:
stat fzlib1g:i386.postinst
File: `fzlib1g:i386.postinst'
Size: 184 Blocks: 8 IO Block: 4096 regular file
Device: 903h/2307d Inode: 23963 Links: 59300
Access: (0750/-rwxr-x---) Uid: ( 104/backuppc) Gid: ( 106/backuppc)
Access: 2014-10-12 02:58:23.000000000 +1100
Modify: 2009-03-09 18:35:31.000000000 +1100
Change: 2014-10-12 03:40:57.000000000 +1100
Birth: -
In fact, some files have over 60000 links.
My /etc/backuppc/config.pl includes this line:
$Conf{HardLinkMax} = '999999';
So, having over 60000 hard links should not be an issue.
Well, now I'm still rather confused... Reading through
BackupPC_backupDuplicate I'm thinking that really the whole v4 is not
relevent, because v3 should have created the link in the cpool (V3
cpool) anyway, and so the question is why did that fail?
Equally, regardless of why the pool file is missing, v4 should be
ensuring that the pool file exists.... That way the problem will at
least get repaired (eg, imagine some issue where some pool files
vanished by mistake, it seems this would never be repaired/discovered),
not that I've tested that.
I'd appreciate any suggestions on how I should proceed.
Should I run my messy code above to "fix" the pool?
Should I simply delete all backups for this host, and then start fresh?
(I'm assuming V4 won't actually download the files since most will still
exist in the pool).
Delete some number of backups to reduce the number of hard links?
Thank you for even reading this far :)
Regards,
Adam
--
Adam Goryachev
Website Managers
www.websitemanagers.com.au
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://p.sf.net/sfu/Zoho
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
