Jim writes: > I initially used cygwin rsync; for the above test, I switched it out for > DeltaCopy's rsync. BOTH VERSIONS had this kind of crappy speed. Both > versions showed hardly any CPU or filesystem usage; they just simply run > slowly for a reason I can't figure out. The network isn't slow (gigabit > ethernet), the checksums aren't taking a long time (it's a brand new > file that doesn't exist on the target so there's nothing to checksum), > the hard drive isn't slow (raid-0 SATA stripe capable of 130GB/s read > speeds) -- it just simply serves data really really slowly.
Rsync on cygwin used to be even worse. Rsync 2.6.1 included a number of changes I made to add I/O buffering, so that the number of system calls and network packets were reduced significantly. Each system call via cygwin has a significant overhead. Without buffering, rsync was sending a huge number of very small ethernet packets. Here's the email (from 7 years ago!). On rsync/cygwin I saw a 3.5x speedup, with one third the cpu usage. I don't know whether or rsync has regressed in some way (I would presume not). But my email describes some of the methodology that could you use to test your current set up. Craig ---------- Forwarded message ---------- To: [email protected] Subject: Rsync performance increase through buffering From: Craig Barratt <[email protected]> Message-ID: <[email protected]> Date: Sun, 08 Dec 2002 23:48:57 -0800 I've been studying the read and write buffering in rsync and it turns out most I/O is done just a couple of bytes at a time. This means there are lots of system calls, and also most network traffic comprises lots of small packets. The behavior is most extreme when sending/receiving file deltas of identical files. The main case where I/O is buffered is writes from the server (when io multiplexing is on). These are usually buffered in 4092 byte chunks with a 4 byte header. However, reading of these packets is usually unbuffered, and writes from the client are generally not buffered. For example: when receiving 1st phase checksums (6 bytes per block), 2 reads are done: one of 4 bytes and one of 2 bytes, meaning there are 4 system calls (select/read/select/read) per 6 bytes of checksum data). One cost of this is some performance, but a significant issue is that unbuffered writes generate very short (and very many) ethernet packets, which means the overhead is quite large on slow network connections. The initial file_list writing is typically buffered, but reading it on the client is not. There are some other unneeded system calls: - One example is that show_progress() calls gettimeofday() even if do_progress is not set. show_progress() is called on every block, so there is an extra system call per (700 byte) block. - Another example is that file_write writes each matching (700 byte) block without buffering, so that's another system call per block. To study this behavior I used rsync-2.5.6cvs and had a benchmark area comprising around 7800 files of total size 530MB. Here are some results doing sends and receives via rsyncd, all on the same machine, with identical source and destination files. In each case --ignore-times (-I) is set, so that every file is processed: - Send test: strace -f rsync -Ir . localhost::test |& wc shows there are about 2,488,775 system calls. - Receive test: strace -f rsync -Ir localhost::test . |& wc shows there are about 1,615,931 system calls. - Rsyncd has a roughly similar numbers of system calls. - Send test from another machine (cygwin/WinXP laptop): tcpdump port 873 |& wc shows there are about 701,111 ethernet packets (many of them only have a 4 byte payload). Since the source and dest files are the same, the send test only wrote 1,738,797 bytes and read 2,139,848 bytes. These results are similar to rsync 2.5.5. Below is a patch to a few files that adds read and write buffering in the places where the I/O was unbuffered, adds buffering to write_file() and removes the unneeded gettimeofday() system call in show_progress(). The results with the patch are: - Send test: 46,835 system calls, versus 2,488,775. - Receive test: 138,367 system calls, versus 1,615,931. - Send test from another machine: 5,255 ethernet packets, versus 701,111. If the tcp/ip/udp/802.3 per-packet overhead is around 60 bytes, that means the base case transfers an extra 42MB of data, even though the useful data is only around 2MB. The absolute running time on the local rsyncd test isn't much different, probably because the test is really disk io limited and system calls on an unloaded linux system are pretty fast. However, on a network test doing a send from cygwin/WinXP to rsyncd on rh-linux the running time improves from about 700 seconds to 215 seconds (with a cpu load of around 17% versus 58%, if you believe cygwin's cpu stats). This is probably an extreme case since the system call penalty in cygwin is high. But I would suspect a significant improvement is possible with a slow network connection, since a lot less data is being sent. Note also that without -I rsync is already very fast, since it skips (most) files based on attributes. With or without this patch the test suite passes except for daemon-gzip-upload. One risk of buffering is the potential for a bug caused by a missing io_flush: deadlock is possible, so try the patch at your own risk... Craig ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ BackupPC-users mailing list [email protected] List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
