It does and in fact almost has to since pool files are stored according to their md5sum. So unless you have either an md5sum collision (extremely unlikely unless creating them intentionally -- as in (number of files)*2^-128 unlikely), you shouldn't have any files in your pool with an underscore in them.
If you have any such files, use BackupPC_zcat to compare their contents. If they are different, then congrats you have (unintentionally) created a blue moon md5sum collision. If the contents are the same, then indeed for some reason de-duplication isn't working. The only thing that I could think of that could possibly cause duplicates is if the compression is set differently on the different backups -- but I'm not sure that would even create a problem. Also, confirm that all your backups are in a v4 pool... Christian Völker wrote at about 09:12:22 +0100 on Saturday, February 11, 2023: > Hi, > > I am using BackupPC now for years. It is really great. Meanwhile I use > v4.4.0 on Debian. > > As far as I understood it os very efficient in storing identical data. > Now I noticed something which let me doubt this. I guess there is an > explanation. So what do I have? > > I have two clients which have a large share. These two (Debian) clients > sync this share on a daily base through rsync (through a third clientC, > but this should not make a difference). On clientA there is a cron job > doing rsync to clientC and on clientB there is a cron job doing rsync > from clientC. So in the end all three hosts have identical data. rsync > command runs through ssh and use "-avH". > > BackupPC itself is only backing up host clientA so far (since months > now). So the data is stored in /var/lib/backuppc. > > Now I added the clientB share to BackupPC and expected the filesystem > usage on /var/lib/backuppc to stay more or less equal after the backupc > of clientB as the data is already stored from clientA. At least after a > while when doing some cleanups. > > Unfortunately, the usage of the pool increased approximately about the > size of the share and has not been dropped since (more than a week now). > > So my questions are: > > * Is there dupe detecion on BackupPC? > * If so, why does my pool size not decrease after a while? > * If by default it has to decrease, is there an explanation why it > does not on my host? > > Thanks a lot! > > > /KNEBB > > > > _______________________________________________ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki: https://github.com/backuppc/backuppc/wiki > Project: https://backuppc.github.io/backuppc/ _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
