On Apr 15, 2008, at 4:54 PM, Tim Hall wrote: > Hi can anyone comment on back jobs with lots > of files effecting transfer time? > > I have 2 big backup jobs which are taking too > long over a WAN link. Would it be advisable to > break the jobs up into many smaller jobs with > fewer files / job? Would I be gaining anything?
How fast are the WAN links at both ends? You may also want to look at latency and any packet loss between the two endpoints. If you are backing up GB of data even a T1 is pretty slow, especially if you have other data going across the same link, it could create an inadvertent "DoS" if you peg the whole connection. If you are using rsync, and especially if memory and / or CPU is a bottlneck, breaking it up into smaller jobs could help by reducing the size of the "list" that rsync has to build and transfer. Having a lot of files will increase the size of the list. What I have done in some cases is to just "mirror" what I want to backup across the link to a "separate drive" at the remote end and then use BackupPC at the remote end to make a "backup" of the mirror. The "mirror" is just a copy of whatever is on the original, but BackupPC provides the "history" of changes over time so you have more than just last night's backup. That also helps for initial set up since I can use e.g. an external USB drive at the local site, do the initial mirror via USB rather then over the WAN link, and then physically take the USB drive to the remote (or maybe in your case ship the drive) and plug it into the server there. Optionally, substitute eSATA / SCSI / iSCSI / IEEE 1394 your connection type of choice for USB. Backing up many GB over T1 will take several hours, especially if you limit bandwidth so that it does not just peg your T1 and put everything else out of commission until it finishes. For e.g 20 GiB * 1024 MiB / GiB = 20480 MiB / 1.544 MiB/s = 13264 s / 3600 s/ hour = 3:41 hours That would be for an optimal point-to-point T1 ignoring bandwidth lost to framing bits, in practice will most likely take quite a bit longer than that, especially if it is through VPN across the internet with possible congestion at routers along the way. > > > would 10 jobs with a 100,000 files each complete > faster then 1 job with 1,000,000 files .... if they > were backing up the same information > > thanks > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save > $100. > Use priority code J8TL2D2. > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > _______________________________________________ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki: http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
