Re: [BackupPC-users] Wildly different speeds for hosts
yet another situation when IO is the enemy. I know most people are most concerned with IO performance on the server, but the client also must be able to keep up or you get .66MB's or something. Raman Gupta wrote: Raman Gupta wrote: I have three hosts configured to backup to my PC. Here are the speeds from the host summary: host 1: 24.77 GB, 14,000 files, 18.78 MB/s (slower WAN link) host 2: 1.27 GB, 4,000 files, 1.89 MB/s (faster WAN link) host 3: 4.82 GB, 190,000 files, 0.66 MB/s (fast LAN link) They all use rsync with the same setup, other than the exclude list. Backups are configured to run one at a time so there is no overlap between them. The speed of host 3 concerns me. Host 3 is by far the beefiest machine, and on the fastest network link of all the hosts, but yet backs up at only 0.66 MB/s (incrementals are even slower). Ok, it seems that the number of files has a large non-linear affect on the performance of BackupPC. I excluded a bunch of stuff from my host 3 backup, and the new stats are: host 3:4.2 GB, 85,000 files, 2.19 MB/s For a file count reduction factor of 2.2, there was a speed increase factor of 3.3. Cheers, Raman - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Wildly different speeds for hosts
Raman Gupta wrote: I have three hosts configured to backup to my PC. Here are the speeds from the host summary: host 1: 24.77 GB, 14,000 files, 18.78 MB/s (slower WAN link) host 2: 1.27 GB, 4,000 files, 1.89 MB/s (faster WAN link) host 3: 4.82 GB, 190,000 files, 0.66 MB/s (fast LAN link) They all use rsync with the same setup, other than the exclude list. Backups are configured to run one at a time so there is no overlap between them. The speed of host 3 concerns me. Host 3 is by far the beefiest machine, and on the fastest network link of all the hosts, but yet backs up at only 0.66 MB/s (incrementals are even slower). Ok, it seems that the number of files has a large non-linear affect on the performance of BackupPC. I excluded a bunch of stuff from my host 3 backup, and the new stats are: host 3:4.2 GB, 85,000 files, 2.19 MB/s For a file count reduction factor of 2.2, there was a speed increase factor of 3.3. Cheers, Raman - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Wildly different speeds for hosts
On Mon, Apr 14, 2008 at 11:21:02AM -0400, Raman Gupta wrote: I have three hosts configured to backup to my PC. Here are the speeds from the host summary: host 1: 24.77 GB, 14,000 files, 18.78 MB/s (slower WAN link) host 2: 1.27 GB, 4,000 files, 1.89 MB/s (faster WAN link) host 3: 4.82 GB, 190,000 files, 0.66 MB/s (fast LAN link) They all use rsync with the same setup, other than the exclude list. Backups are configured to run one at a time so there is no overlap between them. The speed of host 3 concerns me. Host 3 is by far the beefiest machine, and on the fastest network link of all the hosts, but yet backs up at only 0.66 MB/s (incrementals are even slower). Ok, it seems that the number of files has a large non-linear affect on the performance of BackupPC. I excluded a bunch of stuff from my host 3 backup, and the new stats are: host 3:4.2 GB, 85,000 files, 2.19 MB/s For a file count reduction factor of 2.2, there was a speed increase factor of 3.3. I suppose, BackupPC's speed is mainly affected by random access speed of the server's pool storage. I've got hosts with lots of files as well (small ones, mostly) and they take pretty long to back up. Look at I/O utilization of the client during backup - it might be a bottleneck as well. Reading a file linearly is a quite cheap operation: The OS will read-ahead (the RAID probably as well), disk heads don't need to move a lot, metadata fits nicely into the OS' disk cache (and stays there) etc. But if you've got a file system with several millions of files (like the pool) distributed across tens of thousands of directories (like the backup directories below the pc/ directory), things get worse: Lots of random seeking across the disk, cache trashing, I/O waiting etc.. I'm thinking about getting another 2 GB of RAM for my BackupPC server and see whether it improves things. This is an iostat -x -k 60 output during backup runs (2 backups in parallel, 1 pretty fast client, 1 pretty slow with lots of files): avg-cpu: %user %nice %system %iowait %steal %idle 1.890.002.539.490.00 86.09 Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sdb 0.00 0.73 91.64 139.54 1208.53 2105.88 604.27 1052.94 14.34 2.75 11.89 4.31 99.57 (Server is quad-core with 2 GB, 3x500GB RAID5 on Dell PERC5/i - switching to RAID10 would probably improve things a lot as well) Bye, Tino. -- „What we resist, persists.” (Zen saying) www.craniosacralzentrum.de www.forteego.de - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
