Re: [BackupPC-users] Slow Rsync Transfer?
Resolved. After looking at the file list, we found a 102GB log file, rsync doesn't like large files and there are a ton of threads about why. Troubleshooting steps that were taken that actually isolated the issue strace -p $PID (The output look like it was catting the file) lsof -f | grep rsync (and the following to confirm) I hope this helps anybody else who might have this issue. Cheers, ___ Dan Lavu System Administrator - Emptoris, Inc. www.emptoris.com Office: 703.995.6052 - Cell: 703.296.0645 -Original Message- From: Adam Goryachev [mailto:mailingli...@websitemanagers.com.au] Sent: Friday, April 29, 2011 12:46 AM To: backuppc-users@lists.sourceforge.net Subject: Re: [BackupPC-users] Slow Rsync Transfer? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 29/04/11 04:08, Dan Lavu wrote: Gerald, Not the case with me, if you look at the host ras03, you see that the average speed is .92MB/s while other host are significantly faster. It is taking 40 hours to do 110GB, while other hosts are doing it in about an hour. I’m about to patch this box and reboot it, it’s been up for 200+ days and I haven’t had a good backup for over a week now. So any input will be helpful, again thanks in advance. One thing I've seen which can really slow down rsync backups is that a large file with changes will be much slower to backup than a number of small files (of the same total size) with the same amount of changes. I backup disk images, original method was to just backup the image, but this was too slow. New method is: use split to divide the file into a series of 20M or 100M files backup these individual files I also do the same with database exports and other software backup files more than around 100M ... it just backup quicker, and also a failed backup will continue from the most recent chunk (in a full backup) instead of restarting the whole file. Also, timeout is shorter because it is reset after each chunk. Regards, Adam - -- Adam Goryachev Website Managers www.websitemanagers.com.au -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk26QpkACgkQGyoxogrTyiXMlgCgghJ14sMasOdtJi28os6rBj4U GeYAnRxasxrFgpSZ442w0+HKDNHJFsZZ =d8vA -END PGP SIGNATURE- -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow Rsync Transfer?
On 4/29/2011 2:07 PM, Dan Lavu wrote: Resolved. After looking at the file list, we found a 102GB log file, rsync doesn't like large files and there are a ton of threads about why. Troubleshooting steps that were taken that actually isolated the issue strace -p $PID (The output look like it was catting the file) lsof -f | grep rsync (and the following to confirm) I hope this helps anybody else who might have this issue. Glad to hear you found the problem. Stalled transfers tend to be a fairly common issue. Ryan -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Slow Rsync Transfer?
Hi, I've been running BackupPC for a while now and I've been noticing an odd trend, I have 29 servers which are identical in hardware and very similar in configuration. 28 of these servers transfer at 30-40MB/s via Rsync, one host and the database servers transfer between 2-10MB/s, even when ran individually when no other hosts are being backed up. I've checked the IOwait on both the BackupPC server which is a 12 disk raid 10, with 2TB of usable space, 15k RPM drives, it has 5% IO while the host has 0%. I've done simple sftp transfers and they transfer at 20-30MB/s with no issues. Here is a backuppc log on one of the problem hosts, Backup# TypeFilled Level Start Date Duration/mins Age/days Server Backup Path 112 fullyes 0 2/5 19:0662.3 81.6/backuppc/pc/ras03/112 140 fullyes 0 3/5 22:0055.8 53.5/backuppc/pc/ras03/140 154 fullyes 0 3/20 04:35 465.2 39.2/backuppc/pc/ras03/154 161 fullyes 0 3/27 21:42 142.3 31.5/backuppc/pc/ras03/161 167 fullyes 0 4/4 18:00854.0 23.7/backuppc/pc/ras03/167 173 fullyes 0 4/11 18:00 1032.9 16.7/backuppc/pc/ras03/173 174 incrno 1 4/12 18:00 152.2 15.7/backuppc/pc/ras03/174 175 incrno 1 4/13 18:00 1019.5 14.7/backuppc/pc/ras03/175 176 incrno 1 4/14 18:00 1594.4 13.7/backuppc/pc/ras03/176 177 incrno 1 4/15 21:11 2094.5 12.6/backuppc/pc/ras03/177 178 fullyes 0 4/19 03:00 2037.8 9.3 /backuppc/pc/ras03/178 179 incrno 1 4/20 18:16 1147.2 7.7 /backuppc/pc/ras03/179 180 incrno 1 4/21 18:04 1840.7 6.7 /backuppc/pc/ras03/180 181 partial yes 0 4/25 12:32 2389.1 2.9 /backuppc/pc/ras03/181 I've already modified the checksum but it had no impact, '--checksum-seed=32761'. How does backuppc calculate the duration and speed? I know my oracle hosts, I kick off the RMAN prior to pulling the backuppc, so that might explain the slow transfer speeds if it calculated size/time which is inclusive of the RMAN (Database Export) backup. This host (above) boggles me though, these are static files that are being transferred and all other 28 hosts transfer just fine. Thanks in advance for any input or troubleshooting steps. ___ Dan Lavu System Administrator - Emptoris, Inc. www.emptoris.com Office: 703.995.6052 - Cell: 703.296.0645 -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow Rsync Transfer?
- Original Message - Hi, I've been running BackupPC for a while now and I've been noticing an odd trend, I have 29 servers which are identical in hardware and very similar in configuration. 28 of these servers transfer at 30-40MB/s via Rsync, one host and the database servers transfer between 2-10MB/s, even when ran individually when no other hosts are being backed up. I've checked the IOwait on both the BackupPC server which is a 12 disk raid 10, with 2TB of usable space, 15k RPM drives, it has 5% IO while the host has 0%. I've done simple sftp transfers and they transfer at 20-30MB/s with no issues. Here is a backuppc log on one of the problem hosts, Backup# TypeFilled Level Start Date Duration/mins Age/daysServer Backup Path 112 fullyes 0 2/5 19:0662.3 81.6/backuppc/pc/ras03/112 140 fullyes 0 3/5 22:0055.8 53.5/backuppc/pc/ras03/140 154 fullyes 0 3/20 04:35 465.2 39.2/backuppc/pc/ras03/154 161 fullyes 0 3/27 21:42 142.3 31.5/backuppc/pc/ras03/161 167 fullyes 0 4/4 18:00854.0 23.7/backuppc/pc/ras03/167 173 fullyes 0 4/11 18:00 1032.9 16.7/backuppc/pc/ras03/173 174 incrno 1 4/12 18:00 152.2 15.7/backuppc/pc/ras03/174 175 incrno 1 4/13 18:00 1019.5 14.7/backuppc/pc/ras03/175 176 incrno 1 4/14 18:00 1594.4 13.7/backuppc/pc/ras03/176 177 incrno 1 4/15 21:11 2094.5 12.6/backuppc/pc/ras03/177 178 fullyes 0 4/19 03:00 2037.8 9.3 /backuppc/pc/ras03/178 179 incrno 1 4/20 18:16 1147.2 7.7 /backuppc/pc/ras03/179 180 incrno 1 4/21 18:04 1840.7 6.7 /backuppc/pc/ras03/180 181 partial yes 0 4/25 12:32 2389.1 2.9 /backuppc/pc/ras03/181 I've already modified the checksum but it had no impact, '--checksum-seed=32761'. How does backuppc calculate the duration and speed? I know my oracle hosts, I kick off the RMAN prior to pulling the backuppc, so that might explain the slow transfer speeds if it calculated size/time which is inclusive of the RMAN (Database Export) backup. This host (above) boggles me though, these are static files that are being transferred and all other 28 hosts transfer just fine. Thanks in advance for any input or troubleshooting steps. My best speed on Windows and Linux servers via rsync is 21 MB/s, the average is closer to 6 MB/s. Gerald -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow Rsync Transfer?
Gerald, Not the case with me, if you look at the host ras03, you see that the average speed is .92MB/s while other host are significantly faster. It is taking 40 hours to do 110GB, while other hosts are doing it in about an hour. I’m about to patch this box and reboot it, it’s been up for 200+ days and I haven’t had a good backup for over a week now. So any input will be helpful, again thanks in advance. Host User #Full Full Age (days) Full Size (GB) Speed (MB/s)#Incr Incr Age (days) Last Backup (days)State Last attempt ras01 7 3.6 106.97 36.06 6 0.6 0.6 idle idle ras02 7 3.5 122.30 29.25 6 0.5 0.5 idle idle ras03 7 9.4 112.47 0.927 3.0 3.0 backup in progress ras04 8 0.5 105.04 40.48 6 1.5 0.5 idle idle ras05 7 2.7 29.77 17.19 6 0.7 0.7 idle idle ras06 7 4.7 38.23 14.11 6 0.7 0.7 idle idle ras07 7 1.8 134.75 30.72 6 0.8 0.8 idle idle ras08 7 1.7 98.78 24.94 6 0.6 0.6 idle idle ras09 7 1.7 13.38 21.73 6 0.7 0.7 idle idle ras10 7 1.5 162.20 36.92 6 0.5 0.5 idle idle ras11 7 2.5 100.25 23.62 6 0.5 0.5 idle idle ras12 7 1.7 92.03 36.86 6 0.7 0.7 idle idle Cheers, ___ Dan Lavu System Administrator - Emptoris, Inc. www.emptoris.com Office: 703.995.6052 - Cell: 703.296.0645 -Original Message- From: Gerald Brandt [mailto:g...@majentis.com] Sent: Thursday, April 28, 2011 1:51 PM To: General list for user discussion,questions and support Subject: Re: [BackupPC-users] Slow Rsync Transfer? - Original Message - Hi, I've been running BackupPC for a while now and I've been noticing an odd trend, I have 29 servers which are identical in hardware and very similar in configuration. 28 of these servers transfer at 30-40MB/s via Rsync, one host and the database servers transfer between 2-10MB/s, even when ran individually when no other hosts are being backed up. I've checked the IOwait on both the BackupPC server which is a 12 disk raid 10, with 2TB of usable space, 15k RPM drives, it has 5% IO while the host has 0%. I've done simple sftp transfers and they transfer at 20-30MB/s with no issues. Here is a backuppc log on one of the problem hosts, Backup# TypeFilled Level Start Date Duration/mins Age/daysServer Backup Path 112 fullyes 02/5 19:0662.3 81.6/backuppc/pc/ras03/112 140 fullyes 03/5 22:00 55.8 53.5/backuppc/pc/ras03/140 154 fullyes 03/20 04:35 465.2 39.2/backuppc/pc/ras03/154 161 fullyes 0
Re: [BackupPC-users] Slow Rsync Transfer?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 29/04/11 04:08, Dan Lavu wrote: Gerald, Not the case with me, if you look at the host ras03, you see that the average speed is .92MB/s while other host are significantly faster. It is taking 40 hours to do 110GB, while other hosts are doing it in about an hour. I’m about to patch this box and reboot it, it’s been up for 200+ days and I haven’t had a good backup for over a week now. So any input will be helpful, again thanks in advance. One thing I've seen which can really slow down rsync backups is that a large file with changes will be much slower to backup than a number of small files (of the same total size) with the same amount of changes. I backup disk images, original method was to just backup the image, but this was too slow. New method is: use split to divide the file into a series of 20M or 100M files backup these individual files I also do the same with database exports and other software backup files more than around 100M ... it just backup quicker, and also a failed backup will continue from the most recent chunk (in a full backup) instead of restarting the whole file. Also, timeout is shorter because it is reset after each chunk. Regards, Adam - -- Adam Goryachev Website Managers www.websitemanagers.com.au -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk26QpkACgkQGyoxogrTyiXMlgCgghJ14sMasOdtJi28os6rBj4U GeYAnRxasxrFgpSZ442w0+HKDNHJFsZZ =d8vA -END PGP SIGNATURE- -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
