Re: [BackupPC-users] Best Practice to Speedup RSYNC
No not easily, no. The iSCSI disk is from a NAS with 16TB of storage (as was the NFS share) and BackupPC doesn't currently run on the NAS. On 29/09/2023 16:35, Les Mikesell wrote: On Fri, Sep 29, 2023 at 1:46 AM David While via BackupPC-users wrote: Hi Yes I believe that could be the cause. I did some tests with the storage mounted as NFS and then mounted as an ISCSI disk. The ISCSI disk was considerably faster. Can you run the backup against the host of the NFS drive instead of another host mounting it over the network? The way rsync works is that the host where it runs has to read the whole contents of each file in order to use the block checksum algorithm to decide what differences to send. If the drive is mounted over NFS, that complete read will be over a network before it can decide how to reduce network use sending to the rsync partner. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Best Practice to Speedup RSYNC
On Fri, Sep 29, 2023 at 1:46 AM David While via BackupPC-users wrote: > > Hi > > Yes I believe that could be the cause. I did some tests with the storage > mounted as NFS and then mounted as an ISCSI disk. > > The ISCSI disk was considerably faster. Can you run the backup against the host of the NFS drive instead of another host mounting it over the network? The way rsync works is that the host where it runs has to read the whole contents of each file in order to use the block checksum algorithm to decide what differences to send. If the drive is mounted over NFS, that complete read will be over a network before it can decide how to reduce network use sending to the rsync partner. -- Les Mikesell lesmikes...@gmail.com ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Best Practice to Speedup RSYNC
Hi, and thanks for your information. Does anyone testet to cut one big job into smaller ones? Thx Am 29.09.23 um 08:26 schrieb David While via BackupPC-users: Hi Yes I believe that could be the cause. I did some tests with the storage mounted as NFS and then mounted as an ISCSI disk. The ISCSI disk was considerably faster. David While On 29/09/2023 00:05, Robert Trevellyan wrote: "The storage is connected via NFS." ^^^ I suspect this is the main bottleneck. Robert Trevellyan On Thu, Sep 28, 2023 at 5:11 PM wrote: Hi, i use BackupPC on Linux with rsync since a while. One Task as a full backup with 1146496 files and 191547.2MiB of size. This takes 781.5 mins of time. The CompressLevel is setup to 9. The storage is connected via NFS. I use latest BackupPC Version on Debian x64. I asked myself, if there is some potential to boost it up. Its also seems that rsync uses single cpu instead of multi-cpu. May its an option to split the job in smaller jobs so more rsync processes can pallel. Perhaps i should lower the compression level. But i got no idea if that makes a difference. What are youre experiences and tipps to maby boost rsyn backups? Thanks all Taste ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project:https://backuppc.github.io/backuppc/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project:https://backuppc.github.io/backuppc/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Best Practice to Speedup RSYNC
Hi Yes I believe that could be the cause. I did some tests with the storage mounted as NFS and then mounted as an ISCSI disk. The ISCSI disk was considerably faster. David While On 29/09/2023 00:05, Robert Trevellyan wrote: "The storage is connected via NFS." ^^^ I suspect this is the main bottleneck. Robert Trevellyan On Thu, Sep 28, 2023 at 5:11 PM wrote: Hi, i use BackupPC on Linux with rsync since a while. One Task as a full backup with 1146496 files and 191547.2MiB of size. This takes 781.5 mins of time. The CompressLevel is setup to 9. The storage is connected via NFS. I use latest BackupPC Version on Debian x64. I asked myself, if there is some potential to boost it up. Its also seems that rsync uses single cpu instead of multi-cpu. May its an option to split the job in smaller jobs so more rsync processes can pallel. Perhaps i should lower the compression level. But i got no idea if that makes a difference. What are youre experiences and tipps to maby boost rsyn backups? Thanks all Taste ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project:https://backuppc.github.io/backuppc/___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] Best Practice to Speedup RSYNC
"The storage is connected via NFS." ^^^ I suspect this is the main bottleneck. Robert Trevellyan On Thu, Sep 28, 2023 at 5:11 PM wrote: > Hi, > > i use BackupPC on Linux with rsync since a while. One Task as a full > backup with 1146496 files and 191547.2MiB of size. This takes 781.5 mins > of time. The CompressLevel is setup to 9. The storage is connected via > NFS. I use latest BackupPC Version on Debian x64. > > I asked myself, if there is some potential to boost it up. Its also > seems that rsync uses single cpu instead of multi-cpu. May its an option > to split the job in smaller jobs so more rsync processes can pallel. > Perhaps i should lower the compression level. But i got no idea if that > makes a difference. > > What are youre experiences and tipps to maby boost rsyn backups? > > Thanks all > > Taste > > > > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:https://github.com/backuppc/backuppc/wiki > Project: https://backuppc.github.io/backuppc/ > ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
[BackupPC-users] Best Practice to Speedup RSYNC
Hi, i use BackupPC on Linux with rsync since a while. One Task as a full backup with 1146496 files and 191547.2MiB of size. This takes 781.5 mins of time. The CompressLevel is setup to 9. The storage is connected via NFS. I use latest BackupPC Version on Debian x64. I asked myself, if there is some potential to boost it up. Its also seems that rsync uses single cpu instead of multi-cpu. May its an option to split the job in smaller jobs so more rsync processes can pallel. Perhaps i should lower the compression level. But i got no idea if that makes a difference. What are youre experiences and tipps to maby boost rsyn backups? Thanks all Taste ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
