I'm glad your speed has improved. Now that your CPU is not the bottleneck,
you have 2 remaining bottlenecks which are the USB hard drive and the
100Mb/s network. Immediately I suspect that the network is the bottleneck
because of the 6MB/s which is pretty typical of 100Mb/s networking. An
inexpensive network card or switch will compound the issue but really, you
can only expect 6-8MB/s over TCP/IP because of protocol overhead and network
collisions.
At this point, I suggest you fix the networking issue. Now, that doesn't
necessarily mean going gigabit though that would help. I suggest you
compress with rsync. you will likely be able to improve how much data is
pushed over the link though your network interface will still report 6MB/s.
You should see 20-50% low backup times, depending on the compressibility of
the data.
I made the jump to gigabit on my LAN at work primarily because of backups.
I run all cisco equipment so I dropped some fat cash on gigabit but it was
worth it.
good luck
On Jan 7, 2008 10:51 AM, Jinshi <[EMAIL PROTECTED]> wrote:
> Thank you all again for your responses. I setup the backup server on a
> better computer now: Pentium 4 2.4GHz with 760M ram. The system hard
> drive interface is also faster (ATA133, instead of ATA66 on old
> computer). The backup files are still on the same USB drive. The data
> files are on a new computer with Q6600 2.4GHz + 4GB ram running Vista
> x64. Total backup files are about 170GB (over 500k files).All hash files
> are still on the backup server. So there should be no major file
> transfer over network (is this correct?).
>
> I did a full backup via rsyncd. It takes about 1 hour for the client
> computer to prepare the file list before the transfer started. Total
> backup time is a little less than 8 hours and average transfer speed
> more than 6MB/sec. Big boost than before. I think the CPU is making the
> big difference and rsync is doing a lot of calculation. Can someone
> explain in detail what rsync is doing? I do not use any compression.
>
> 8 hours is long but tolerable for once a week. I suppose incremental
> backup will be faster. I don't think I will do any upgrade on the hard
> drive/usb/network since I believe the CPU/memory is the limiting step.
> Any thought on fine tune /configure the BackupPC to make it faster?
>
> Thanks.
>
> PS: the USB drive still shows as 40.000MB/s in dmesg. Is this a FreeBSD
> thing? I suppose it should be 60MB for usb2.0 (480Mbps). This may not be
> important here since the file transfer speed is far below 40MB/s.
>
>
> Les Mikesell wrote:
> > Jinshi wrote:
> >> Thank you all for your reply. Apparently everyone consider this is too
> >> slow since the difference between usb2.0 and usb1.1 is quite big. But I
> >> do tested all other usb ports. The dmesg says the transfer speed is
> >> either 40.000MB or 1.000MB/s. So, sorry everyone, I still insist I have
> >> usb2.0 here :)
> >>
> >> I didn't reply sooner because I want to wait and see how long the
> backup
> >> is. Now, total file about 170GB (I moved some files from other
> computer.
> >> Those files were backed up before, so the hash files are available on
> >> the backup computer). There is almost no new files (details below). And
> >> full backup by SMB transfer takes 32 hours (only 7.3MB new files), by
> >> rsyncd transfer takes 26 hours (1.8GB new files since my wife uploaded
> >> more photos during SMB backup time).
> >
> > Was the rsync run a full or incremental? A mostly unchanged incremental
> > should be much faster than a full because the full does a block checksum
> > comparison even on files where the timestamp and length match.
> >
> >> So, rsyncd helps, but not a lot, just like Serge predicted. Average
> >> transfer speed is 1.5 and 1.8 MB/sec, respectively.
> >>
> >> Now, back to my original question: where is the bottleneck? I am going
> >> to upgrade the computer to P4. If that still doesn't help much, I don't
> >> know what to do next?!
> >
> > Rsync transfers the entire directory listing before starting so there is
> > a certain amount of RAM needed per file - if you have less the server
> > might start swapping and slow down. Depending on your directory layout,
> > it might be possible to break the target up into several smaller chunks
> > to help with this. Also, a slightly more extreme approach would be to
> > make the subdirectory runs appear to be different hosts using the alias
> > feature. That way you could stagger the full and incremental runs so
> > you don't have to complete a full in one day.
> >
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
>
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
