3.0.0pre2 performance and assembler warning
In testing 3.0.0pre1 and now 3.0.0pre2 I have noticed that the rate of data transfer is ~25% slower than 2.6.9. I am using rsync to mirror 2 filesystems from a single host. On average for large files ( 4gb) under 3.0.0pre2 the transfer of ~75MB/sec. WIth 2.6.9 the average is ~105MB/sec. I am compiling on a ia64 based system running SLES10 SP1 and see this assembler warnings when compiling with gcc 4.1.2. Are these of any concern? ~/rsync/rsync-3.0.0pre2 make gcc -std=gnu99 -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -o mkrounding -I. ./mkrounding.c ./mkrounding rounding.h Rounding file_extras in multiples of 2 (EXTRA_LEN=4, FILE_STRUCT_LEN=24) gcc -std=gnu99 -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c flist.c -o flist.o /tmp/ccC7MQuD.s: Assembler messages: /tmp/ccC7MQuD.s:2993: Warning: Use of 'mov' may violate WAW dependency 'GR%, % in 1 - 127' (impliedf), specific resource number is 14 /tmp/ccC7MQuD.s:2991: Warning: This is the location of the conflicting usage gcc -std=gnu99 -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c rsync.c -o rsync.o -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --detect-renamed question
Matt, That was too quick! I think --trust-move is a really good thing and I'll test it out soon. Some thoughts on all this now that I've had my caffeine this morning. It would take one very crafty user to delete a file and create one with the same name, mtime, and size. The only issue I can see is if there were 2 files with the same name, mtime, and size but different data. Highly unlikely but still possible, right? Let's call these files fileA and fileB. If fileA is deleted and fileB is copied to another directory, what happens? Rsync would hard link the fileA as the new fileB when using --trust-move. We end up with fileA and fileB on the destination with fileB and fileB on the source. So, how do we fix this situation? Is there a way to check for duplicate entries? If rsync checks if the file it's about to hard link is a non-unique file, (same name, mtime, size as another file) then it should copy from the source fileB instead of hard linking from the deleted fileA. Does this make sense? It would require rsync to have a complete scan of the source prior to doing anything. This should help those situations where someone does an upper level directory move with lots of files and data underneath. I recall someone else was asking about this on the list. Greg On Oct 12, 2007, at 6:43 PM, Matt McCutchen wrote: On 10/12/07, Greg Siekas [EMAIL PROTECTED] wrote: The other option I thought of was to only do the move when the mtime, size, and filename match. Not really a 'detect-renamed' but a 'detected-moved' type operation. That's a good idea, and easy to implement too! I have improved the patch (attached) to provide separate --trust-rename and --trust-move options. Wayne, please consider adding this to patches/ . Matt trust-rename.diff -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Disable checksumming to improve local performance?
Is it possible to disable checksumming? I'm using rsync with -W, whole file, so if a file has changed I don't want to just transfer the changes. The source and destination or local filesystems. I've noticed that performance in 3.0.6 is slower than 2.6.9. 3.0.6 is at ~90MB/sec and 2.6.9 is at ~107MB/sec. I'm looking for anything that can help to improve transfer rates on large files. thanks, Greg -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync local performance
Wayne, Transferring an 8gb file using rsync between a network (10GbE) mounted filesystem and local disk. rsync-2.6.9 - 88-95 MB/sec rsync-3.0.6 - 62-72 MB/sec rsync-3.1.0 - 86-90 MB/sec Doing a cp of the file yields 140-160MB/sec. It appears the IO code improvements in 3.1 have definitely made a difference over the 3.0 code base. Greg -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync local performance
Has anyone compiled rsync with other newer compilers like Intel 11.1? Does this break anything? My quick test shows rsync-3.1.0 performance jumps to ~120MB/sec. Greg On Nov 13, 2009, at 10:44 AM, Greg Siekas wrote: Wayne, Transferring an 8gb file using rsync between a network (10GbE) mounted filesystem and local disk. rsync-2.6.9 - 88-95 MB/sec rsync-3.0.6 - 62-72 MB/sec rsync-3.1.0 - 86-90 MB/sec Doing a cp of the file yields 140-160MB/sec. It appears the IO code improvements in 3.1 have definitely made a difference over the 3.0 code base. Greg -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Why is IO_BUFFER_SIZE set to 4092?
I'm curious as to why IO_BUFFER_SIZE is only set to 4092 in rsync.h? This seems very small. Is there any harm in increasing it? If not, what is the proper way to determine what is should be set to? Greg -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and many files
40 files a second seems very slow. Are you sure the majority of the time is generating the file list and determine what's changed? How many of the millions of files are changed? On modern hardware I see 1000's of files per second when scanning for changed files. On Jun 6, 2011, at 12:39 PM, Steven Levine stev...@earthlink.net wrote: In f992406d6e81b54dbb33210217fe7afd07f80...@exchange1.mtb.netclusive.de, on 06/06/11 at 12:04 PM, Cliff Simon cliff.si...@netclusive.com said: Hi, We are using rsync via rsnapshot, but this is not elementary. It is used to backup many (above 100 servers) and works very well. Now there is one server with many (several millions) files. The files are not very big, so the complete backup is about 500 GB. Now my problem is, that the backup needs about 14 hours - the most time is to generate the filelist and check whether the files are new/changed or not. My rsync-command is: /usr/bin/rsync -a --bwlimit=9000 --delete --numeric-ids --relative --delete-excluded --exclude=/some/pathes/ --rsh=/usr/bin/ssh --link-dest=/dest.path/daily.1/ root@192.x.x.x:/path.to.backup/ Do you have an idea to reduce the backup time? A bit of math says 2*10^6 / 14 hours is about 40 files/second. How fast do you think rsync should be and how does this compare to backups on your other servers? Are you sure the it is not the hardware that is limiting the rsync's performance? Based on my knowledge of the rsync sources, I believe the file list generation algorithms are pretty efficient. There is quite a bit of code in the code path, but it's hard to avoid this given the number of options available to control the sync process. Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: threaded rsync to improve performance over long distance ?
With the SSH-HPN you can achieve over 200MB/sec. Depends on your latency and packet drops but you should certainly try a modified ssh/scp with rsync. Also are you using compression with rsync -z? That's an area that could be multi threaded. On Jan 20, 2013, at 3:25 PM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 You can use the HPN enhancement patches for ssh. They do make a big difference: http://www.psc.edu/index.php/hpn-ssh On 01/20/13 18:22, Markus Moeller wrote: Hi Kevin, but rsync uses ssh doesn't it ? Are there any tuning options for buffers to deal with the tcp SYN SYN-ACK delay ? Thank you Markus Kevin Korb k...@sanitarium.net wrote in message news:50fc5e6f.1000...@sanitarium.net... rsync does not use scp in any way. On 01/20/13 14:47, Markus Moeller wrote: I try to rsync a lot of data over a long distance AU to UK and get a slow data rate because of the known slow scp performance over long distances. Would it be possible with rsync to run multiple scp process at the same time ? Thank you Markus -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin KorbPhone:(407) 252-6853 Systems AdministratorInternet: FutureQuest, Inc.ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page:http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlD8fOkACgkQVKC1jlbQAQcqTwCeMsxo3MMyfFx3omGpIu5ELDuM V64AoLdq5ak55CPIIjVk7AHK6U+0XTbN =pu+/ -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
