Re: [BackupPC-users] Conf{PoolNightlyDigestCheckPercent} on btrfs and zfs
Hi Guillermo,
I agree. BackupPC's RefCnt/Fsck is a must IMHO.
A main difference between PoolNightlyDigestCheck and btrfs' builtin
checksumming is that the nightly digest checks proactively scan the data and
will tell you about broken files as soon as they break (well, as soon as the
next check is done after the broke). Btrfs will only complain on retrieval;
perhaps soon, perhaps never. So you need a way to make sure that your archived
data is actually read every now and then, and compared to the checksums for
consistency.
For btrfs, this is done by the "scrub" operation. Which is recommended to be
done regularly.
In my experience, it's slightly lower on the CPU load than BackupPC's check,
but the actual load should be on I/O to the disk, not the CPU. Default CRC32 is
somewhat weaker than the MD5s used by BackupPC, but this is a concern for only
pool collisions, not for data rot. Newer btrfs' features include, e.g., xxhash
checksums, which are not only faster but also more collision resistant than MD5.
I personally use btrfs scrub exclusively (which also check's my non-BackupPC
files on that disk) and disabled NightlyDigestCheck. However, there is one
significant drawback that might or might not be an issue for you: btrfs scrub
is all-or-nothing; it always scans the entire partition and, to be frank, the
"idle priority scheduler" does not play nicely on every system. In contrast,
BackupPC's nightly checks are more clever in that you can scan parts of your
pool every night, to distribute the load on your system.
So, IMHO, the best approach is to have a separate partition/device exclusively
for your pool, use NightlyDigestCheck there, and *disable* (or not enable)
regular btrfs scrub. (You can keep checksumming active nevertheless, it's
extremely low overhead.)
A scrub does oh-so-slightly more, as it also checks metadata checksums and can
repair corrupted blocks if good copies are available - but problems will also
be detected by a BackupPC nightly scan, and corrective action can be taken.
For any of those, make sure you receive notifications by mail, monitoring, or
LEDs above your pillow in case there are errors. Spending a lot of time on
checksumming, reading SMART data, health checks etc. isn't going to help unless
you know about the results.
For ZFS, no clue.
Best,
Alex
On 5/4/21 6:38 PM, Guillermo Rozas wrote:
> One ensures against file system bit rot, the other ensures backup file
> consistency.
>
>
> I would say $Conf{PoolNightlyDigestCheckPercent} = 1 is also a check for bit
> rot, as the only thing it does is to read the file, re-calculate the md5
> checksum, and compares it with its name (which is the md5 calculated at the
> time of writing). It actually says it in the help, "This is check if there
> has been any server file system corruption."
>
> What controls the consistency of the backup are
> $Conf{PoolSizeNightlyUpdatePeriod} and specially $Conf{RefCntFsck}.
>
> Regards,
> Guillermo
>
>
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:https://github.com/backuppc/backuppc/wiki
> Project: https://backuppc.github.io/backuppc/
>
smime.p7s
Description: S/MIME Cryptographic Signature
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Conf{PoolNightlyDigestCheckPercent} on btrfs and zfs
>
> One ensures against file system bit rot, the other ensures backup file
> consistency.
>
I would say $Conf{PoolNightlyDigestCheckPercent} = 1 is also a check for
bit rot, as the only thing it does is to read the file, re-calculate the
md5 checksum, and compares it with its name (which is the md5 calculated at
the time of writing). It actually says it in the help, "This is check if
there has been any server file system corruption."
What controls the consistency of the backup are
$Conf{PoolSizeNightlyUpdatePeriod} and specially $Conf{RefCntFsck}.
Regards,
Guillermo
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Conf{PoolNightlyDigestCheckPercent} on btrfs and zfs
On 2021-05-04 00:45, Ghislain Adnet wrote: No, because that's like turning off the airbags on your car because you already have seatbelts. thanks for your answer, i am trying to understand more about this. If this check is to prevent bitrot or disk corruption of an existing file, and if rsync is sure the file is ok, and then that ZFS keep the checksum and guarantee that the file did not change (assuming raidz1 at least of course) then, at the end what could be the case where this check help find issue ? -- cordialement, Ghislain One ensures against file system bit rot, the other ensures backup file consistency. Cheers! mph ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Conf{PoolNightlyDigestCheckPercent} on btrfs and zfs
No, because that's like turning off the airbags on your car because you already have seatbelts. thanks for your answer, i am trying to understand more about this. If this check is to prevent bitrot or disk corruption of an existing file, and if rsync is sure the file is ok, and then that ZFS keep the checksum and guarantee that the file did not change (assuming raidz1 at least of course) then, at the end what could be the case where this check help find issue ? -- cordialement, Ghislain -- cordialement, Ghislain ADNET. AQUEOS. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Conf{PoolNightlyDigestCheckPercent} on btrfs and zfs
On 2021-04-30 00:09, Ghislain Adnet wrote: hi, Should this setting put to 0 on ZFS or BTRFS as the filesystem allready do checksumming and will detect issues ? No, because that's like turning off the airbags on your car because you already have seatbelts. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
[BackupPC-users] Conf{PoolNightlyDigestCheckPercent} on btrfs and zfs
hi,
Should this setting put to 0 on ZFS or BTRFS as the filesystem allready do
checksumming and will detect issues ?
Or is it something différent ?
$Conf{PoolNightlyDigestCheckPercent} = 1;
Integrity check the pool files by confirming the md5 digest of the contents
matches their file name. Because the pool is very large, only check a small
random percentage of the pool files each night.
This is check if there has been any server file system corruption.
The default value of 1% means approximately 30% of the pool files will be
checked each month, although the actual number will be a bit less since some
files might be checked more than once in that time. If BackupPC_nightly takes
too long, you could reduce this value.
--
cordialement,
Ghislain
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/
