100gb of 4-40MB files sounds like my home PC full of digital photos I've taken. It backs up to a linux PC right beside it with rsync. I don't really call it that big a project for rsync. Big things for rsync are millions of files. At 100mbps, it takes a few seconds to build the list. I use the -W option to copy the whole file instead of mess with incremental on a fast network. The linux PC running rsync only has 768MB, athlon 1700, and runs X, so it's nothing special.
It will take a while over the VPN to build the file list and start transferring, but it shouldn't be hard on the computer resouces. I used to back the whole system up over a 512kbps link to our datacenter, but don't anymore. That was reliable but obviously slow. On Fri, Mar 03, 2006 at 08:02:55AM +0100, [EMAIL PROTECTED] wrote: > // I wonder if this message has been posted, so I sent it again // > > Hello, > > I'm quite a n00b on rsync stuff but I went to the website, read > FAQ/how-to, Google and more, I setup my own rsync server and clients: > everything works fine :-D > > I'm preparing a plan for a production mode in my company: we need to > mirror around 100GB of data trough a special VPN internet line 2MB > symmetric. > The first time, the data will be transferred by a media such as a HD. > Next, each night, we will try to update clients from the master server. > It should be around 500MB to 3GB, no so much in comparison of the > original size of data. > I discovered "rsync" use a lot of CPU and RAM to run "checksums" on > file that have to be synchronised. I need an opinion about my situation: > > > So: each night, from 0:00am to maximum 7:00am, the server will have to > check the 100Go of files and see what files have been modified, then, > upload them to the clients. Each file is around 4MB to 40MB in average. > > I would like to know your opinion about this situation: > - Should I setup a strong dual CPU computer dedicated to calculate this > whole stuff? > - What about the memory I should install? > - Is there any bandwidth used during the checksums computation? Mine is > quite limited. > - I know the client computer will have to check files too; Disk I/O > will be the most used. I think this computer will have NFS mount from a > "datacenter" computer with a GB LAN card, I wonder it will be enough... > > I'm quite scared of the amount of data to check before synchronise > clients, and how long it will take. To finish shortly, what do YOU > think? Any advices? > > > Thanks, > > Johan > -- > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- /* Jason Philbrook | Midcoast Internet Solutions - Internet Access, KB1IOJ | Hosting, and TCP-IP Networks for Midcoast Maine http://f64.nu/ | http://www.midcoast.com/ */ -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html