Re: [BackupPC-users] How to check pool?

2024-02-21 Thread G.W. Haywood

Hi there,

On Wed, 21 Feb 2024, Alexander Kobel wrote:

February 18, 2024 at 9:52 AM, Christian V?lker wrote:
> 
> I have a large v4 pool (not cpool) (1.6TB) running on top of btrfs.


For the record, that isn't a large pool for BackupPC to manage.


> Is there any chance to perform a pool check from BackupPC to verify
> all data in the pool is still ok?  I am getting some checksum errors
> from btrfs and I want to know if the backed up data is still fine.

`dmesg` will show you where the checksum errors occur. ...
...
Unfortunately, it is non-trivial and time-consuming to find where a
file/hash from the pool is referenced in backups. ...


If you have filesystem errors then you really can't depend on it.  If
this were my filesystem I'd completely replace it.  But then there are
other people, not just me, depending on those backups.

Last time we had a serious filesystem error like this was about twenty
years ago, when unfortunately I was wooed by the promises of ReiserFS.
It lasted about a week, until it unrecoverably ate a test filesystem.
The provided fsck made it worse, I ditched it and went back to ext3.

If you need your backups, don't accept anything less than perfection.

Do the supplied btrfs tools not fix the errors?

https://en.wikipedia.org/wiki/Btrfs#Checking_and_recovery
https://en.wikipedia.org/wiki/Btrfs#Checksum_tree_and_scrubbing

Note that for some of the recovery operations you're expected to be,
or to be advised by, "a developer or an experienced user" of btrfs.
If you are not, perhaps btrfs isn't for you.  I wouldn't yet trust it
with my backups.

Is this USB-connected hardware?  If so, be very careful with it; I'd
be inclined to run at least two backup systems, perhaps running them
alternately.  I'm not sure that USB is safe for backups, and I'm not
sure that btrfs is a good choice anyway.

Do you have uninterruptible power on the backup system?

--

73,
Ged.


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/


Re: [BackupPC-users] How to check pool?

2024-02-20 Thread Alexander Kobel
Hi Christian,

`dmesg` will show you where the checksum errors occur. Can be combined with a 
`btrfs scrub` to get a full report over your entire volume.
If you see entries in the BackupPC pool or pc directories, that'll generally 
mean that the affected file(s) are broken and can't be repaired from within the 
backup system. Short of deep-diving on the individuals bits and hexes on the 
disk for partial recovery, not a lot you can do about it; so if possible, 
delete them and have them re-back-uped next time.
Note that without deleting them, v4 will not look at the contents of the files, 
IIUC - given the filename/overall hash, it will just assume that the pool file 
and the existing file on the client match and not retransfer. Only exception is 
if the file changed or freshly appeared on the client, in which case it will be 
compared against the pool file. So you'll usually not see a checksum error 
during backup, only on the nightly scans where the contents of the pool files 
are read and their hash is re-calculated.

Unfortunately, it is non-trivial and time-consuming to find where a file/hash 
from the pool is referenced in backups. If I remember correctly, you need to 
walk the attrib files for that.
I (think it was me who) wrote a utility for that ages ago; I can't even 
remember. But it being based on zsh looks like it's not part of the BackupPC 
distro, and could stem from me... Horrible coding, slow, documentation=code, 
certainly not industry strength, all support my authorship. ;-)
To be executed in the BackupPC topdir, as far as I remember; it should roughly 
print you where a file with a certain hash appears in your pool. But I guess my 
use case was similar to yours. Timestamp is from 2019 and I didn't seem to have 
used it since then. Which probably means it's been tested and written only with 
v3; not sure whether it still works for v4 without changes, but you can try.

Please find it attached, and check whether it helps you something. No promises, 
it might eat your data and your cat (though it shouldn't).


Cheers,
Alex


February 18, 2024 at 9:52 AM, "Christian Völker via BackupPC-users" 
mailto:backuppc-users@lists.sourceforge.net?to=%22Christian%20V%C3%B6lker%20via%20BackupPC-users%22%20%3Cbackuppc-users%40lists.sourceforge.net%3E>>
 wrote:

Hi all,


I have a large v4 pool (not cpool) (1.6TB) running on top of btrfs.


Is there any chance to perform a pool check from BackupPC to verify all 
data in the pool is still ok?


I am getting some checksum errors from btrfs and I want to know if the 
backed up data is still fine.


Thanks!


/KNEBB






___

BackupPC-users mailing list

BackupPC-users@lists.sourceforge.net
List:
https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:
https://github.com/backuppc/backuppc/wiki
Project:
https://backuppc.github.io/backuppc/



#!/bin/zsh

hash=$1
TOPDIR=$(pwd)

hashaa=$(printf %02x $((0x${hash[1,2]} & 0xfe)))

setopt nullglob

hosts=()
for i in pc/*/; do
host=${i:t}
find $TOPDIR/pc/$host/refCnt -maxdepth 1 -name poolCnt.'[01]'.$hashaa -type 
f -print0 2>/dev/null | while read -d$'\0' poolCnt; do
BackupPC_poolCntPrint $poolCnt | grep $hash > /dev/null && hosts+=$host
done
done

echo matching hosts: $hosts

typeset -A host_nums
for host in $hosts; do
nums=()
for i in pc/$host/[0-9]*/; do
num=${i:t}
find $TOPDIR/pc/$host/$num/refCnt -maxdepth 1 -name 
poolCnt.'[01]'.$hashaa -type f -print0 2>/dev/null | while read -d$'\0' 
poolCnt; do
count=$(BackupPC_poolCntPrint $poolCnt | grep $hash | wc -l)
[[ $count -gt 0 ]] && {
nums+=$num
host_nums[$host,$num]=$count
}
done
done
done

echo matching backups:
for host_num count in ${(kv)host_nums}; do
echo "  $host_num ($count occurences)" | sed -e 's/,/#/'
done | sort

echo searching backups:
for host_num count in ${(kv)host_nums}; do
host=${host_num%,*}
num=${host_num##*,}
echo "  searching $host#$num"
find $TOPDIR/pc/$host/$num -maxdepth 1 -name 'f%2f*' -type d -print0 
2>/dev/null | while read -d$'\0' sharePath; do
share=${sharePath:t}
share=${share[2,-1]}
share=${share:gs/%10/\\n}
share=${share:gs/%13/\\r}
share=${share:gs/%2f/\/}
share=${share:gs/%25/%}
echo "searching $host:$share#$num"
BackupPC_ls -R $sharePath | grep $hash
done
done
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/


[BackupPC-users] How to check pool?

2024-02-18 Thread Christian Völker via BackupPC-users

Hi all,

I have a large v4 pool (not cpool) (1.6TB) running on top of btrfs.

Is there any chance to perform a pool check from BackupPC to verify all 
data in the pool is still ok?


I am getting some checksum errors from btrfs and I want to know if the 
backed up data is still fine.


Thanks!

/KNEBB





___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/