Re: [BackupPC-users] Files not taken from pool even when they are identical

[Jean-Louis Biasini via BackupPC-users]

Thank you both very much for your answer an explanation. after checking 
with the ls -i command the two files are indeed on the same inode.


Le 01/07/2019 à 04:22, Adam Goryachev a écrit :



On 28/6/19 9:46 pm, Jean-Louis Biasini via BackupPC-users wrote:

Hi all,

I have a working installation on a centos7 server. I'm backing up 30+ 
linux server hosts with rsync. Since I was a bit surprised by the 
growing space taken on the backup server, I started to investigate on 
the pooling mechanism. So I started to create simple identical text 
files on 2 servers to see if the file was showed as already in pool 
while backing up the second server after having back up the first. 
First I just created 2 times the same text file (ie same content, 
same md5sum, same permission, same selinux but different creating 
time) second I created it on the first server then rsync it to the 
second to have it absolutely identical (rsync -aAXv). In both case 
the file is showed as created by the second server's backup. Then I 
tried a bigger file created with fallocate -l 2M test.img and rsynced 
it the same way. In all case my file is created again on the second 
backup. I also checked the hard link limit that I increased from 
32000 to 64000 (ext4 file-system here) with no improvement. Am I 
missing something?



You didn't mention which version of backuppc you are using, so I can't 
be sure, but from memory, with BPC 3.x that is the correct/expected 
behaviour. However, while the file is transferred from the second host 
to the BPC server, it *will* get pooled correctly, so there will only 
be a single on-disk version of the file. You can check this by 
confirming the hard link entry on the BPC server for the 2 hosts in 
question.


eg:

ls -i /var/lib/backuppc/pc//123/f%2f/ftest.txt 
/var/lib/backuppc/pc//124/f%2f/ftest.txt


If the inode number is the same for both files, then they are 
correctly de-duplicated/pooled.



This might be the same for BPC 4.x, although I recall (and have 
confirmed ages ago) that BPC 4.x doesn't transfer the file if the 
duplicate file is located on the same host. So, it may still transfer 
the file in your example (different hosts), but again should not be 
duplicated in the pool (I don't know how to verify this on BPC 4.x).


Maybe something like this:

/usr/lib/backuppc/bin/BackupPC_attribPrint 
/var/lib/backuppc/pc//123/f%2f/attrib_*


Check the inode entry, and do the same for your second host, make sure 
the inode entry is the same, and I assume this would mean that the 
pooling is working correctly.


PS, just because the xferlog shows a file is transferred, doesn't mean 
it isn't pooled after BPC receives the full file, and confirms the 
content is a match for the other file it already has.


Regards,
Adam




___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Files not taken from pool even when they are identical

[Adam Goryachev]

On 28/6/19 9:46 pm, Jean-Louis Biasini via BackupPC-users wrote:

Hi all,

I have a working installation on a centos7 server. I'm backing up 30+ 
linux server hosts with rsync. Since I was a bit surprised by the 
growing space taken on the backup server, I started to investigate on 
the pooling mechanism. So I started to create simple identical text 
files on 2 servers to see if the file was showed as already in pool 
while backing up the second server after having back up the first. 
First I just created 2 times the same text file (ie same content, same 
md5sum, same permission, same selinux but different creating time) 
second I created it on the first server then rsync it to the second to 
have it absolutely identical (rsync -aAXv). In both case the file is 
showed as created by the second server's backup. Then I tried a bigger 
file created with fallocate -l 2M test.img and rsynced it the same 
way. In all case my file is created again on the second backup. I also 
checked the hard link limit that I increased from 32000 to 64000 (ext4 
file-system here) with no improvement. Am I missing something?



You didn't mention which version of backuppc you are using, so I can't 
be sure, but from memory, with BPC 3.x that is the correct/expected 
behaviour. However, while the file is transferred from the second host 
to the BPC server, it *will* get pooled correctly, so there will only be 
a single on-disk version of the file. You can check this by confirming 
the hard link entry on the BPC server for the 2 hosts in question.


eg:

ls -i /var/lib/backuppc/pc//123/f%2f/ftest.txt 
/var/lib/backuppc/pc//124/f%2f/ftest.txt


If the inode number is the same for both files, then they are correctly 
de-duplicated/pooled.



This might be the same for BPC 4.x, although I recall (and have 
confirmed ages ago) that BPC 4.x doesn't transfer the file if the 
duplicate file is located on the same host. So, it may still transfer 
the file in your example (different hosts), but again should not be 
duplicated in the pool (I don't know how to verify this on BPC 4.x).


Maybe something like this:

/usr/lib/backuppc/bin/BackupPC_attribPrint 
/var/lib/backuppc/pc//123/f%2f/attrib_*


Check the inode entry, and do the same for your second host, make sure 
the inode entry is the same, and I assume this would mean that the 
pooling is working correctly.


PS, just because the xferlog shows a file is transferred, doesn't mean 
it isn't pooled after BPC receives the full file, and confirms the 
content is a match for the other file it already has.


Regards,
Adam

--
Adam Goryachev Website Managers www.websitemanagers.com.au
--
The information in this e-mail is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this e-mail by anyone else
is unauthorised. If you are not the intended recipient, any disclosure,
copying, distribution or any action taken or omitted to be taken in reliance
on it, is prohibited and may be unlawful. If you have received this message
in error, please notify us immediately. Please also destroy and delete the
message from your computer. Viruses - Any loss/damage incurred by receiving
this email is not the sender's responsibility.


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Files not taken from pool even when they are identical

[Jean-Louis Biasini via BackupPC-users]

Hi Craig!

Thanks for your answer. I'm running the version from epel repository, 
tagged as: 3.3.1-5.el7


jean-louis


Le 30/06/2019 à 19:58, Craig Barratt a écrit :

Jean-Louis,

What version of BackupPC are you running?

Craig

On Fri, Jun 28, 2019 at 5:11 AM Jean-Louis Biasini via BackupPC-users 
> wrote:


Hi all,

I have a working installation on a centos7 server. I'm backing up 30+
linux server hosts with rsync. Since I was a bit surprised by the
growing space taken on the backup server, I started to investigate on
the pooling mechanism. So I started to create simple identical text
files on 2 servers to see if the file was showed as already in pool
while backing up the second server after having back up the first.
First
I just created 2 times the same text file (ie same content, same
md5sum,
same permission, same selinux but different creating time) second I
created it on the first server then rsync it to the second to have it
absolutely identical (rsync -aAXv). In both case the file is
showed as
created by the second server's backup. Then I tried a bigger file
created with fallocate -l 2M test.img and rsynced it the same way. In
all case my file is created again on the second backup. I also
checked
the hard link limit that I increased from 32000 to 64000 (ext4
file-system here) with no improvement. Am I missing something?

Many thanks

jean-louis



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net

List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Files not taken from pool even when they are identical

[Craig Barratt via BackupPC-users]

Jean-Louis,

What version of BackupPC are you running?

Craig

On Fri, Jun 28, 2019 at 5:11 AM Jean-Louis Biasini via BackupPC-users <
backuppc-users@lists.sourceforge.net> wrote:

> Hi all,
>
> I have a working installation on a centos7 server. I'm backing up 30+
> linux server hosts with rsync. Since I was a bit surprised by the
> growing space taken on the backup server, I started to investigate on
> the pooling mechanism. So I started to create simple identical text
> files on 2 servers to see if the file was showed as already in pool
> while backing up the second server after having back up the first. First
> I just created 2 times the same text file (ie same content, same md5sum,
> same permission, same selinux but different creating time) second I
> created it on the first server then rsync it to the second to have it
> absolutely identical (rsync -aAXv). In both case the file is showed as
> created by the second server's backup. Then I tried a bigger file
> created with fallocate -l 2M test.img and rsynced it the same way. In
> all case my file is created again on the second backup. I also checked
> the hard link limit that I increased from 32000 to 64000 (ext4
> file-system here) with no improvement. Am I missing something?
>
> Many thanks
>
> jean-louis
>
>
>
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[BackupPC-users] Files not taken from pool even when they are identical

[Jean-Louis Biasini via BackupPC-users]

Hi all,

I have a working installation on a centos7 server. I'm backing up 30+ 
linux server hosts with rsync. Since I was a bit surprised by the 
growing space taken on the backup server, I started to investigate on 
the pooling mechanism. So I started to create simple identical text 
files on 2 servers to see if the file was showed as already in pool 
while backing up the second server after having back up the first. First 
I just created 2 times the same text file (ie same content, same md5sum, 
same permission, same selinux but different creating time) second I 
created it on the first server then rsync it to the second to have it 
absolutely identical (rsync -aAXv). In both case the file is showed as 
created by the second server's backup. Then I tried a bigger file 
created with fallocate -l 2M test.img and rsynced it the same way. In 
all case my file is created again on the second backup. I also checked 
the hard link limit that I increased from 32000 to 64000 (ext4 
file-system here) with no improvement. Am I missing something?


Many thanks

jean-louis



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[BackupPC-users] Files not taken from pool even when they are identical

[Jean-Louis Biasini via BackupPC-users]

Hi all,

I have a working installation on a centos7 server. I'm backing up 30+ 
linux server hosts with rsync. Since I was a bit surprised by the 
growing space taken on the backup server, I started to investigate on 
the pooling mechanism. So I started to create simple identical text 
files on 2 servers to see if the file was showed as already in pool 
while backing up the second server after having back up the first. First 
I just created 2 times the same text file (ie same content, same md5sum, 
same permission, same selinux but different creating time) second I 
created it on the first server then rsync it to the second to have it 
absolutely identical (rsync -aAXv). In both case the file is showed as 
created by the second server's backup. Then I tried a bigger file 
created with fallocate -l 2M test.img and rsynced it the same way. In 
all case my file is created again on the second backup. I also checked 
the hard link limit that I increased from 32000 to 64000 (ext4 
file-system here) with no improvement. Am I missing something?


Many thanks

jean-louis




___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/