Re: [BackupPC-users] Slow Rsync Transfer?
Resolved. After looking at the file list, we found a 102GB log file, rsync doesn't like large files and there are a ton of threads about why. Troubleshooting steps that were taken that actually isolated the issue strace -p $PID (The output look like it was catting the file) lsof -f | grep rsync (and the following to confirm) I hope this helps anybody else who might have this issue. Cheers, ___ Dan Lavu System Administrator - Emptoris, Inc. www.emptoris.com Office: 703.995.6052 - Cell: 703.296.0645 -Original Message- From: Adam Goryachev [mailto:mailingli...@websitemanagers.com.au] Sent: Friday, April 29, 2011 12:46 AM To: backuppc-users@lists.sourceforge.net Subject: Re: [BackupPC-users] Slow Rsync Transfer? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 29/04/11 04:08, Dan Lavu wrote: Gerald, Not the case with me, if you look at the host ras03, you see that the average speed is .92MB/s while other host are significantly faster. It is taking 40 hours to do 110GB, while other hosts are doing it in about an hour. I’m about to patch this box and reboot it, it’s been up for 200+ days and I haven’t had a good backup for over a week now. So any input will be helpful, again thanks in advance. One thing I've seen which can really slow down rsync backups is that a large file with changes will be much slower to backup than a number of small files (of the same total size) with the same amount of changes. I backup disk images, original method was to just backup the image, but this was too slow. New method is: use split to divide the file into a series of 20M or 100M files backup these individual files I also do the same with database exports and other software backup files more than around 100M ... it just backup quicker, and also a failed backup will continue from the most recent chunk (in a full backup) instead of restarting the whole file. Also, timeout is shorter because it is reset after each chunk. Regards, Adam - -- Adam Goryachev Website Managers www.websitemanagers.com.au -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk26QpkACgkQGyoxogrTyiXMlgCgghJ14sMasOdtJi28os6rBj4U GeYAnRxasxrFgpSZ442w0+HKDNHJFsZZ =d8vA -END PGP SIGNATURE- -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Slow Rsync Transfer?
Hi, I've been running BackupPC for a while now and I've been noticing an odd trend, I have 29 servers which are identical in hardware and very similar in configuration. 28 of these servers transfer at 30-40MB/s via Rsync, one host and the database servers transfer between 2-10MB/s, even when ran individually when no other hosts are being backed up. I've checked the IOwait on both the BackupPC server which is a 12 disk raid 10, with 2TB of usable space, 15k RPM drives, it has 5% IO while the host has 0%. I've done simple sftp transfers and they transfer at 20-30MB/s with no issues. Here is a backuppc log on one of the problem hosts, Backup# TypeFilled Level Start Date Duration/mins Age/days Server Backup Path 112 fullyes 0 2/5 19:0662.3 81.6/backuppc/pc/ras03/112 140 fullyes 0 3/5 22:0055.8 53.5/backuppc/pc/ras03/140 154 fullyes 0 3/20 04:35 465.2 39.2/backuppc/pc/ras03/154 161 fullyes 0 3/27 21:42 142.3 31.5/backuppc/pc/ras03/161 167 fullyes 0 4/4 18:00854.0 23.7/backuppc/pc/ras03/167 173 fullyes 0 4/11 18:00 1032.9 16.7/backuppc/pc/ras03/173 174 incrno 1 4/12 18:00 152.2 15.7/backuppc/pc/ras03/174 175 incrno 1 4/13 18:00 1019.5 14.7/backuppc/pc/ras03/175 176 incrno 1 4/14 18:00 1594.4 13.7/backuppc/pc/ras03/176 177 incrno 1 4/15 21:11 2094.5 12.6/backuppc/pc/ras03/177 178 fullyes 0 4/19 03:00 2037.8 9.3 /backuppc/pc/ras03/178 179 incrno 1 4/20 18:16 1147.2 7.7 /backuppc/pc/ras03/179 180 incrno 1 4/21 18:04 1840.7 6.7 /backuppc/pc/ras03/180 181 partial yes 0 4/25 12:32 2389.1 2.9 /backuppc/pc/ras03/181 I've already modified the checksum but it had no impact, '--checksum-seed=32761'. How does backuppc calculate the duration and speed? I know my oracle hosts, I kick off the RMAN prior to pulling the backuppc, so that might explain the slow transfer speeds if it calculated size/time which is inclusive of the RMAN (Database Export) backup. This host (above) boggles me though, these are static files that are being transferred and all other 28 hosts transfer just fine. Thanks in advance for any input or troubleshooting steps. ___ Dan Lavu System Administrator - Emptoris, Inc. www.emptoris.com Office: 703.995.6052 - Cell: 703.296.0645 -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Slow Rsync Transfer?
Gerald, Not the case with me, if you look at the host ras03, you see that the average speed is .92MB/s while other host are significantly faster. It is taking 40 hours to do 110GB, while other hosts are doing it in about an hour. I’m about to patch this box and reboot it, it’s been up for 200+ days and I haven’t had a good backup for over a week now. So any input will be helpful, again thanks in advance. Host User #Full Full Age (days) Full Size (GB) Speed (MB/s)#Incr Incr Age (days) Last Backup (days)State Last attempt ras01 7 3.6 106.97 36.06 6 0.6 0.6 idle idle ras02 7 3.5 122.30 29.25 6 0.5 0.5 idle idle ras03 7 9.4 112.47 0.927 3.0 3.0 backup in progress ras04 8 0.5 105.04 40.48 6 1.5 0.5 idle idle ras05 7 2.7 29.77 17.19 6 0.7 0.7 idle idle ras06 7 4.7 38.23 14.11 6 0.7 0.7 idle idle ras07 7 1.8 134.75 30.72 6 0.8 0.8 idle idle ras08 7 1.7 98.78 24.94 6 0.6 0.6 idle idle ras09 7 1.7 13.38 21.73 6 0.7 0.7 idle idle ras10 7 1.5 162.20 36.92 6 0.5 0.5 idle idle ras11 7 2.5 100.25 23.62 6 0.5 0.5 idle idle ras12 7 1.7 92.03 36.86 6 0.7 0.7 idle idle Cheers, ___ Dan Lavu System Administrator - Emptoris, Inc. www.emptoris.com Office: 703.995.6052 - Cell: 703.296.0645 -Original Message- From: Gerald Brandt [mailto:g...@majentis.com] Sent: Thursday, April 28, 2011 1:51 PM To: General list for user discussion,questions and support Subject: Re: [BackupPC-users] Slow Rsync Transfer? - Original Message - Hi, I've been running BackupPC for a while now and I've been noticing an odd trend, I have 29 servers which are identical in hardware and very similar in configuration. 28 of these servers transfer at 30-40MB/s via Rsync, one host and the database servers transfer between 2-10MB/s, even when ran individually when no other hosts are being backed up. I've checked the IOwait on both the BackupPC server which is a 12 disk raid 10, with 2TB of usable space, 15k RPM drives, it has 5% IO while the host has 0%. I've done simple sftp transfers and they transfer at 20-30MB/s with no issues. Here is a backuppc log on one of the problem hosts, Backup# TypeFilled Level Start Date Duration/mins Age/daysServer Backup Path 112 fullyes 02/5 19:0662.3 81.6/backuppc/pc/ras03/112 140 fullyes 03/5 22:00 55.8 53.5/backuppc/pc/ras03/140 154 fullyes 03/20 04:35 465.2 39.2/backuppc/pc/ras03/154 161 fullyes 0
Re: [BackupPC-users] Compress::Zlib
Oh, for anybody who might have an issue with Compress::Zlib, here is a solution that worked for me. http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing- lists-3/backuppc-21/backuppc-cant-find-compress-zlib-after-recent-update -on-cen-106280/ Dan Lavu System Administrator Office: 703.995.6052 Cell: 703.296.0645 dan.l...@rivermine.com www.rivermine.com This message may contain information that is proprietary to Rivermine, its customers, and partners. If you have received it in error, please notify Rivermine immediately. -Original Message- From: Dan Lavu [mailto:dan.l...@rivermine.com] Sent: Monday, September 27, 2010 3:41 PM To: backuppc-users@lists.sourceforge.net Subject: [BackupPC-users] Compress::Zlib So I patched my systems due to a recent Redhat/Centos 64bit root exploit and now Compress::Zlib is broken. ### 2010-09-27 13:24:34 User dan.lavu requested backup of server01 ( server01) 2010-09-27 13:24:34 Backup failed on server01 (can't find Compress::Zlib) 2010-09-27 14:00:00 Next wakeup is 2010-09-27 15:00:00 ### I've tried running CPAN, and reinstalling Compress::Zlib but no avail, it refuses to compile. I've checked out the latest DAG repos for an rpm and I did an upgrade but it still cannot find the library. ### Bareword Compress::Zlib::zlib_version not allowed while strict subs in use at t/cz-14gzopen.t line 34. Bareword ZLIB_VERSION not allowed while strict subs in use at t/cz-14gzopen.t line 34. Bareword Z_FINISH not allowed while strict subs in use at t/cz-14gzopen.t line 108. Bareword Z_STREAM_END not allowed while strict subs in use at t/cz-14gzopen.t line 121. Bareword Z_STREAM_END not allowed while strict subs in use at t/cz-14gzopen.t line 122. Bareword Z_STREAM_ERROR not allowed while strict subs in use at t/cz-14gzopen.t line 469. Bareword Z_STREAM_ERROR not allowed while strict subs in use at t/cz-14gzopen.t line 484. Execution of t/cz-14gzopen.t aborted due to compilation errors. # Looks like you planned 255 tests but only ran 2. # Looks like your test died just after 2. t/cz-14gzopen...dubious Test returned status 255 (wstat 65280, 0xff00) DIED. FAILED tests 1, 3-255 Failed 254/255 tests, 0.39% okay t/globmapperok Failed Test Stat Wstat Total Fail Failed List of Failed --- t/000prereq.t2 512302 6.67% 3 30 t/01misc.t 1 256 1181 0.85% 3 t/cz-01version.t 255 65280 23 150.00% 1-2 t/cz-03zlib-v1.t 255 65280 456 907 198.90% 1 4-456 t/cz-05examples.t 255 65280?? ?? % ?? t/cz-06gzsetp.t255 65280?? ?? % ?? t/cz-08encoding.t 255 6528029 57 196.55% 1-29 t/cz-14gzopen.t255 65280 255 507 198.82% 1 3-255 3 tests and 29 subtests skipped. Failed 8/85 test scripts, 90.59% okay. 742/52174 subtests failed, 98.58% okay. make: *** [test_dynamic] Error 255 /usr/bin/make test -- NOT OK Running make install make test had returned bad status, won't install without force ### This is on CentOS 5.5, is anybody else having this issue? -- Best Regards, Dan Lavu Systems Administrator Rivermine Software 703 995 6052 (o) 703 296 0645 (c) -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Compress::Zlib
So I patched my systems due to a recent Redhat/Centos 64bit root exploit and now Compress::Zlib is broken. ### 2010-09-27 13:24:34 User dan.lavu requested backup of server01 ( server01) 2010-09-27 13:24:34 Backup failed on server01 (can't find Compress::Zlib) 2010-09-27 14:00:00 Next wakeup is 2010-09-27 15:00:00 ### I've tried running CPAN, and reinstalling Compress::Zlib but no avail, it refuses to compile. I've checked out the latest DAG repos for an rpm and I did an upgrade but it still cannot find the library. ### Bareword Compress::Zlib::zlib_version not allowed while strict subs in use at t/cz-14gzopen.t line 34. Bareword ZLIB_VERSION not allowed while strict subs in use at t/cz-14gzopen.t line 34. Bareword Z_FINISH not allowed while strict subs in use at t/cz-14gzopen.t line 108. Bareword Z_STREAM_END not allowed while strict subs in use at t/cz-14gzopen.t line 121. Bareword Z_STREAM_END not allowed while strict subs in use at t/cz-14gzopen.t line 122. Bareword Z_STREAM_ERROR not allowed while strict subs in use at t/cz-14gzopen.t line 469. Bareword Z_STREAM_ERROR not allowed while strict subs in use at t/cz-14gzopen.t line 484. Execution of t/cz-14gzopen.t aborted due to compilation errors. # Looks like you planned 255 tests but only ran 2. # Looks like your test died just after 2. t/cz-14gzopen...dubious Test returned status 255 (wstat 65280, 0xff00) DIED. FAILED tests 1, 3-255 Failed 254/255 tests, 0.39% okay t/globmapperok Failed Test Stat Wstat Total Fail Failed List of Failed --- t/000prereq.t2 512302 6.67% 3 30 t/01misc.t 1 256 1181 0.85% 3 t/cz-01version.t 255 65280 23 150.00% 1-2 t/cz-03zlib-v1.t 255 65280 456 907 198.90% 1 4-456 t/cz-05examples.t 255 65280?? ?? % ?? t/cz-06gzsetp.t255 65280?? ?? % ?? t/cz-08encoding.t 255 6528029 57 196.55% 1-29 t/cz-14gzopen.t255 65280 255 507 198.82% 1 3-255 3 tests and 29 subtests skipped. Failed 8/85 test scripts, 90.59% okay. 742/52174 subtests failed, 98.58% okay. make: *** [test_dynamic] Error 255 /usr/bin/make test -- NOT OK Running make install make test had returned bad status, won't install without force ### This is on CentOS 5.5, is anybody else having this issue? -- Best Regards, Dan Lavu Systems Administrator Rivermine Software 703 995 6052 (o) 703 296 0645 (c) -- Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
