Re: [BackupPC-users] BackupPC misconfiguration Rsync network usage
On Wed, 2009-01-07 at 21:20 +1100, Adam Goryachev wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 William McKee wrote: I use VMware on a co-lo server which has 3 guestts that all get backed up by BackupPC. I could identify that the host was transmitting massive amounts of data (130Gb) which appeared to be coming from one of the three guests. However, I couldn't figure out which guest was pushing out the excessive data. I went through the usual log files without much luck. I then checked the ifconfig output which all looked normal inside the hosts. Once I finally looked at the BackupPC logs for the guest server, I realized what was happening and corrected the issue by removing my bad entry. I also added --bwlimit to the RsyncArgs setting in config.pl to control maxing out my bandwidth. However, this all took longer than I'd have liked. I'm stumped as to why the data transmitted off of the guest did not show up in the ifconfig output. I know that the guest is sending data via rsync based on the logs. However it's not showing up in the ifconfig stats (see below). Is this due to the way that rsync works? I was sending about 450Mb of data every 1-2 hrs from 8pm - 6am (I can send the logs if that would be of any help). I've included below the ifconfig outputs for the host (massive TX bytes) and the guest (normal TX bytes). I would have expected a corresponding amount of TX bytes for the guest. Thanks for any insight. I would suspect vmware has something to do with that. Try creating traffic with any other tool, and it likely won't be counted in the way you think it should as well. Another option is perhaps the counters wrapped due to the amount of data... so if they wrapped recently, then the values will be very small, even though a huge amount of data has been transmitted. There is nothing special that rsync does to cause it's bandwidth not to be counted normally (AFAIK). Regards, Adam I think Adam is spot on. AFAIK, ifconfig receives its byte data from the proc subsystem and on 32 bit systems will wrap at 4 gigs. Regards, Royden -- Check out the new SourceForge.net Marketplace. It is the best place to buy or sell services for just about anything Open Source. http://p.sf.net/sfu/Xq1LFB ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] BackupPC misconfiguration Rsync network usage
Hi all, This evening I tracked down a configuration error that was causing a bandwidth spike due to a misconfiguration of BackupPC (v2.1.2). I set the IncrPeriod to 0.00 thinking that no incrementals would get run. Boy was that wrong! Instead, it ran incrementals one after another during off-peak hours. That spiked my bandwidth with my hosting provider which sent me searching for the culprit. Because of the holidays, I had forgotten about the edit of the IncrPeriod so wasn't sure what was causing the spike. Thus I went digging through my logs and such to try to identify the culprit. I use VMware on a co-lo server which has 3 guestts that all get backed up by BackupPC. I could identify that the host was transmitting massive amounts of data (130Gb) which appeared to be coming from one of the three guests. However, I couldn't figure out which guest was pushing out the excessive data. I went through the usual log files without much luck. I then checked the ifconfig output which all looked normal inside the hosts. Once I finally looked at the BackupPC logs for the guest server, I realized what was happening and corrected the issue by removing my bad entry. I also added --bwlimit to the RsyncArgs setting in config.pl to control maxing out my bandwidth. However, this all took longer than I'd have liked. I'm stumped as to why the data transmitted off of the guest did not show up in the ifconfig output. I know that the guest is sending data via rsync based on the logs. However it's not showing up in the ifconfig stats (see below). Is this due to the way that rsync works? I was sending about 450Mb of data every 1-2 hrs from 8pm - 6am (I can send the logs if that would be of any help). I've included below the ifconfig outputs for the host (massive TX bytes) and the guest (normal TX bytes). I would have expected a corresponding amount of TX bytes for the guest. Thanks for any insight. Cheers, William Output of ifconfig on host (atlas) eth0 Link encap:Ethernet HWaddr 00:17:A4:3F:C3:B5 inet addr:64.132.42.194 Bcast:64.132.42.207 Mask:255.255.255.240 inet6 addr: fe80::217:a4ff:fe3f:c3b5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:67509721 errors:0 dropped:0 overruns:0 frame:3 TX packets:102403892 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6915124969 (6.4 GiB) TX bytes:139582421865 (129.9 GiB) Interrupt:16 Output of ifconfig on guest (wg75) eth0 Link encap:Ethernet HWaddr 00:0c:29:2a:5f:cd inet addr:192.168.233.25 Bcast:192.168.233.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe2a:5fcd/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:26307738 errors:0 dropped:0 overruns:0 frame:0 TX packets:42720081 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2111065627 (1.9 GB) TX bytes:2438535854 (2.2 GB) Interrupt:17 Base address:0x1400 -- Knowmad Technologies - Open Source Web Solutions W: http://www.knowmad.com | E: will...@knowmad.com | P: 704.343.9330 -- Check out the new SourceForge.net Marketplace. It is the best place to buy or sell services for just about anything Open Source. http://p.sf.net/sfu/Xq1LFB ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC misconfiguration Rsync network usage
William, Is the guest machine multi homed - with multiple network interface cards. Linux binds a IP Address to entire OS rather than to a specific interface unlike some of the other UNIX flavors that binds it only to the interface. If it is really multi-homed, you might get a clue by looking at the ifconfig output of all eth? interfaces. There might be another eth? interface that would have a corresponding increase of packets that you are expecting... HTH Vasan On Wed, Jan 7, 2009 at 12:56 PM, William McKee will...@knowmad.com wrote: Hi all, This evening I tracked down a configuration error that was causing a bandwidth spike due to a misconfiguration of BackupPC (v2.1.2). I set the IncrPeriod to 0.00 thinking that no incrementals would get run. Boy was that wrong! Instead, it ran incrementals one after another during off-peak hours. That spiked my bandwidth with my hosting provider which sent me searching for the culprit. Because of the holidays, I had forgotten about the edit of the IncrPeriod so wasn't sure what was causing the spike. Thus I went digging through my logs and such to try to identify the culprit. I use VMware on a co-lo server which has 3 guestts that all get backed up by BackupPC. I could identify that the host was transmitting massive amounts of data (130Gb) which appeared to be coming from one of the three guests. However, I couldn't figure out which guest was pushing out the excessive data. I went through the usual log files without much luck. I then checked the ifconfig output which all looked normal inside the hosts. Once I finally looked at the BackupPC logs for the guest server, I realized what was happening and corrected the issue by removing my bad entry. I also added --bwlimit to the RsyncArgs setting in config.pl to control maxing out my bandwidth. However, this all took longer than I'd have liked. I'm stumped as to why the data transmitted off of the guest did not show up in the ifconfig output. I know that the guest is sending data via rsync based on the logs. However it's not showing up in the ifconfig stats (see below). Is this due to the way that rsync works? I was sending about 450Mb of data every 1-2 hrs from 8pm - 6am (I can send the logs if that would be of any help). I've included below the ifconfig outputs for the host (massive TX bytes) and the guest (normal TX bytes). I would have expected a corresponding amount of TX bytes for the guest. Thanks for any insight. Cheers, William Output of ifconfig on host (atlas) eth0 Link encap:Ethernet HWaddr 00:17:A4:3F:C3:B5 inet addr:64.132.42.194 Bcast:64.132.42.207 Mask:255.255.255.240 inet6 addr: fe80::217:a4ff:fe3f:c3b5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:67509721 errors:0 dropped:0 overruns:0 frame:3 TX packets:102403892 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6915124969 (6.4 GiB) TX bytes:139582421865 (129.9 GiB) Interrupt:16 Output of ifconfig on guest (wg75) eth0 Link encap:Ethernet HWaddr 00:0c:29:2a:5f:cd inet addr:192.168.233.25 Bcast:192.168.233.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe2a:5fcd/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:26307738 errors:0 dropped:0 overruns:0 frame:0 TX packets:42720081 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2111065627 (1.9 GB) TX bytes:2438535854 (2.2 GB) Interrupt:17 Base address:0x1400 -- Knowmad Technologies - Open Source Web Solutions W: http://www.knowmad.com | E: will...@knowmad.com | P: 704.343.9330 -- Check out the new SourceForge.net Marketplace. It is the best place to buy or sell services for just about anything Open Source. http://p.sf.net/sfu/Xq1LFB ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Check out the new SourceForge.net Marketplace. It is the best place to buy or sell services for just about anything Open Source. http://p.sf.net/sfu/Xq1LFB___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC misconfiguration Rsync network usage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 William McKee wrote: I use VMware on a co-lo server which has 3 guestts that all get backed up by BackupPC. I could identify that the host was transmitting massive amounts of data (130Gb) which appeared to be coming from one of the three guests. However, I couldn't figure out which guest was pushing out the excessive data. I went through the usual log files without much luck. I then checked the ifconfig output which all looked normal inside the hosts. Once I finally looked at the BackupPC logs for the guest server, I realized what was happening and corrected the issue by removing my bad entry. I also added --bwlimit to the RsyncArgs setting in config.pl to control maxing out my bandwidth. However, this all took longer than I'd have liked. I'm stumped as to why the data transmitted off of the guest did not show up in the ifconfig output. I know that the guest is sending data via rsync based on the logs. However it's not showing up in the ifconfig stats (see below). Is this due to the way that rsync works? I was sending about 450Mb of data every 1-2 hrs from 8pm - 6am (I can send the logs if that would be of any help). I've included below the ifconfig outputs for the host (massive TX bytes) and the guest (normal TX bytes). I would have expected a corresponding amount of TX bytes for the guest. Thanks for any insight. I would suspect vmware has something to do with that. Try creating traffic with any other tool, and it likely won't be counted in the way you think it should as well. Another option is perhaps the counters wrapped due to the amount of data... so if they wrapped recently, then the values will be very small, even though a huge amount of data has been transmitted. There is nothing special that rsync does to cause it's bandwidth not to be counted normally (AFAIK). Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAklkgdsACgkQGyoxogrTyiXTTACguBsJwvOLXCi/+Mgs2ML6JYV5 mOcAoJ61TAmlpkE9j/6tyu9PkHxsnDp3 =elRS -END PGP SIGNATURE- -- Check out the new SourceForge.net Marketplace. It is the best place to buy or sell services for just about anything Open Source. http://p.sf.net/sfu/Xq1LFB ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
