I discovered what the issue was. It appears you cannot feed just the full filename for the tar-based incremental backup. BackupPC does not appear to be creating the directory structure if missing. It creates the files on the filesystem and they will show up in the "Browse Backups" if you specify the full path to it. However, the linkage in the tree appears to be missing... so you can't use "Browse Backup" at the root path and navigate your way to them, and BackupPC_ls suffers the same.
My solution: in my pre-cmd script that I populate /mnt/backuppc/pc/$hostIP/incl.lst, for each file, loop and call dirname and write the output, until it reaches my base path. When finished, sort and uniq the file. The sorting will re-write the file so that it is more like a standard directory walker. Now BackupPC's "Browse Backups" can be used to navigate from the root to the new files and they show up in BackupPC_ls now. I also added -no-recursion to my TarIncr arguments to avoid excessive inclusion. Strangely they still show up as "Xfer errors" but there are no actual errors. The logs indicate they are "other" category files. I can live with that. The whole reason for this set up is for consistency between database backups and the incremental backups, where files may have been created after the database snapshot was taken. Basing it on timestamps alone in my use case is unreliable, because we may end up with either missing files that the database has records referencing, or we could end up with files that exist but the database has no records of. Restoring in either case would lead to bad data, so the solution to that problem was to be explicit on what gets included in an incremental backup. Hope this helps someone out. From: Michael Martinek <mmarti...@kardin.com> Sent: Monday, December 7, 2020 4:08 PM To: backuppc-users@lists.sourceforge.net Subject: [BackupPC-users] Incremental files missing when supplied Greetings, I have a database that contains file references, so when a backup is performed it will first use mariabackup to do an online copy of a database, stage it locally, and then generate a list of files based on a SQL query. I then compare that list to that of BackupPC_ls using the latest backup. This seems to work great, and the goal is ultimately that each backup will contain all the related files that the database contains-so a recovery state would not have missing DB entries or missing files. This is done with a "pre-cmd" script that also builds out an "inclusion list" used in the dump process: TarClientPath: /usr/bin/gtar TarClientCmd: /usr/bin/env LC_ALL=C /usr/bin/sudo $tarPath -c -v -C $shareName -totals TarFullArgs: -f - $fileList TarIncrArgs: -T /mnt/backup/pc/$hostIP/incl.lst The logs do not indicate any errors. It is picking up the files in the incl.lst, and I can browse to it if I paste the full path in "Browse backups". However, the directories/files backed up in this way are not showing in the "Browse Backups" if I navigate the directories by clicking links, nor do they show up through BackupPC_ls. They exist on disk with proper permissions, and if I re-run an incremental backup the log indicates the files are "same"-so BackupPC is definitely seeing the files. I'm wondering if anyone has encountered a similar issue. Basically, it backing up the file with the -T <filename> approach. I have tried various formats in the incl.lst (ie: including the share prefix, excluding it, etc). Currently it contains: -C/mnt/dev/files ./files/5ac800f4d5cb23a13dca1420da7681f98b042618/d2d82809bb778ad2598aad392580997618707440/fbb4bcc11d345602eb73e3e0bde0eab327d1007b/e0693a02de54f31ae0a88d1daf45eec9e0308e0e/3/TestFile Note the -C is probably redundant, as it is passed in the TarClientCmd already. It was just a troubleshooting effort to ensure it wasn't an escaping issue with $shareName. The log for this incremental test: Successfully dumped database. Running: /usr/bin/env LC_ALL=C /usr/bin/sudo /usr/bin/gtar -c -v -C /mnt/dev/files --totals -T /mnt/backup/pc/127.0.0.1/incl.lst incr backup started back to 2020-12-07 18:16:53 (backup #0) for directory /mnt/dev/files Xfer PIDs are now 1058574,1058573 [ skipped 1 lines ] tarExtract: /usr/share/BackupPC/bin/BackupPC_tarExtract: got Full = tarExtract: /usr/share/BackupPC/bin/BackupPC_tarExtract starting... (XferLogLevel = 5) tarExtract: Got file './files/5ac800f4d5cb23a13dca1420da7681f98b042618/d2d82809bb778ad2598aad392580997618707440/fbb4bcc11d345602eb73e3e0bde0eab327d1007b/e0693a02de54f31ae0a88d1daf45eec9e0308e0e/3/TestFile', mode 0644, size 5.95968e+06, type 0 [ skipped 1 lines ] readOutput: sysread returns 0 and got EOF tarExtract: No match... adding /mnt/backup/cpool/1058573.0.0 to pool (digest = 801dd115621e4d06005b04cb8baf74ea) [ skipped 1 lines ] tarExtract: File files/5ac800f4d5cb23a13dca1420da7681f98b042618/d2d82809bb778ad2598aad392580997618707440/fbb4bcc11d345602eb73e3e0bde0eab327d1007b/e0693a02de54f31ae0a88d1daf45eec9e0308e0e/3/TestFile: old digest , new digest 801dd115621e4d06005b04cb8baf74ea tarExtract: No match... adding /mnt/backup/cpool/1058573.1.0 to pool (digest = 825fcb7dc04d17990ece7126d3a03403) tarExtract: bpc_attrib_dirWrite: created new attrib file /mnt/backup/pc/127.0.0.1/1/f%2fmnt%2fdev%2ffiles/ffiles/f5ac800f4d5cb23a13dca1420da7681f98b042618/fd2d82809bb778ad2598aad392580997618707440/ffbb4bcc11d345602eb73e3e0bde0eab327d1007b/fe0693a02de54f31ae0a88d1daf45eec9e0308e0e/f3/attrib_825fcb7dc04d17990ece7126d3a03403 tarExtract: Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 1 filesTotal, 5959680 sizeTotal, 1 filesNew, 5959680 sizeNew, 523999 sizeNewComp, 96160 inodeLast Stat of the file that BackupPC created: File: /mnt/backup/pc/127.0.0.1/1/f%2fmnt%2fdev%2ffiles/ffiles/f5ac800f4d5cb23a13dca1420da7681f98b042618/fd2d82809bb778ad2598aad392580997618707440/ffbb4bcc11d345602eb73e3e0bde0eab327d1007b/fe0693a02de54f31ae0a88d1daf45eec9e0308e0e/f3/attrib_825fcb7dc04d17990ece7126d3a03403 Size: 0 Blocks: 0 IO Block: 1048576 regular empty file Device: 32h/50d Inode: 41944691 Links: 1 Access: (0640/-rw-r-----) Uid: ( 993/backuppc) Gid: ( 989/backuppc) Context: system_u:object_r:nfs_t:s0 Access: 2020-12-07 21:08:38.598385821 +0000 Modify: 2020-12-07 21:08:38.598385821 +0000 Change: 2020-12-07 21:08:38.600385949 +0000 Birth: - I have been at this now for a few days and I have pretty much run out of ideas. If anyone can assist or has an idea then I would greatly appreciate a hand. Thanks in advance, Michael Martinek
_______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
