Re: How to change using port from 873 to 22
On Wed 14 Nov 2007, ohtsu masahiro wrote: The native rsync protocol over port 873 is currently used in my server. I would like to use not port 873 but port 22). How should I change ? all I have to change is command option? rsync -rp --rsh -- rsync -rp --ssh ??? Have you actually looked an the manpage? I would think it's pretty clear you need the --port=22 option. But why use port 22? Won't that already be in use by ssh? And what's using port 873, which is reserved for rsync? Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Problem with rsync recent file logic ?
Hello, I have 2 servers I'm synchronizing using rsync, I have a situation where I : 1. rsync from rnd-dev2 to rnd-dev1 2. change the rsynched file on rnd-dev1 3. rsync from rnd-dev2 to rnd-dev1 again 4. File gets overridden on rnd-dev1 over though it has newer change time then file on rnd-dev2. here is the bug(?) reproduction: [EMAIL PROTECTED] test_rsync]# rsync --version rsync version 2.6.3 protocol version 28 Copyright (C) 1996-2004 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, IPv6, 64-bit system inums, 64-bit internal inums [EMAIL PROTECTED] maxim]# mkdir /tmp/test_rsync [EMAIL PROTECTED] maxim]# mkdir /tmp/test_rsync [EMAIL PROTECTED] test_rsync]# dd if=/dev/urandom of=maxim_file.dat count=1024 [EMAIL PROTECTED] test_rsync]# md5sum maxim_file.dat 517cbaefb385f64823179b253844fff0 maxim_file.dat [EMAIL PROTECTED] test_rsync]# stat maxim_file.dat File: `maxim_file.dat' Size: 524288 Blocks: 1032 IO Block: 4096 regular file Device: fd00h/64768dInode: 1081370 Links: 1 Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) Access: 2007-11-15 12:25:30.0 +0200 Modify: 2007-11-15 12:24:37.0 +0200 Change: 2007-11-15 12:24:37.0 +0200 [EMAIL PROTECTED] test_rsync]# rsync --timeout 20 -avvr /tmp/test_rsync/ -e ssh [EMAIL PROTECTED]:/tmp/test_rsync/ opening connection using ssh -l root rnd-dev1 rsync --server -vvlogDtpr --timeout=20 . /tmp/test_rsync/ building file list ... [sender] expand file_list to 131072 bytes, did move done delta transmission enabled ./ maxim_file.dat total: matches=0 tag_hits=0 false_alarms=0 data=524288 sent 524456 bytes received 40 bytes 1048992.00 bytes/sec total size is 524288 speedup is 1.00 [EMAIL PROTECTED] test_rsync]# md5sum maxim_file.dat 517cbaefb385f64823179b253844fff0 maxim_file.dat [EMAIL PROTECTED] test_rsync]# md5sum maxim_file.dat 517cbaefb385f64823179b253844fff0 maxim_file.dat # So far so good ## Here comes the BUG part ## [EMAIL PROTECTED] test_rsync]# dd if=/dev/urandom of=maxim_file.dat count=1024 1024+0 records in 1024+0 records out [EMAIL PROTECTED] test_rsync]# md5sum maxim_file.dat c780e88b1bf60e5cb31ead145c5c3b39 maxim_file.dat [EMAIL PROTECTED] test_rsync]# stat maxim_file.dat File: `maxim_file.dat' Size: 524288 Blocks: 1032 IO Block: 4096 regular file Device: fd00h/64768dInode: 261636 Links: 1 Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) Access: 2007-11-15 12:33:32.0 +0200 Modify: 2007-11-15 12:33:37.0 +0200 Change: 2007-11-15 12:33:37.0 +0200 [EMAIL PROTECTED] test_rsync]# stat maxim_file.dat File: `maxim_file.dat' Size: 524288 Blocks: 1032 IO Block: 4096 regular file Device: fd00h/64768dInode: 1081370 Links: 1 Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) Access: 2007-11-15 12:35:06.0 +0200 Modify: 2007-11-15 12:24:37.0 +0200 Change: 2007-11-15 12:24:37.0 +0200 [EMAIL PROTECTED] test_rsync]# rsync --timeout 20 -avvvr /tmp/test_rsync/ -e ssh [EMAIL PROTECTED]:/tmp/test_rsync/ opening connection using ssh -l root rnd-dev1 rsync --server -vvvlogDtpr --timeout=20 . /tmp/test_rsync/ building file list ... [sender] make_file(.,*,2) [sender] expand file_list to 131072 bytes, did move [sender] make_file(maxim_file.dat,*,2) [sender] popping per-dir .cvsignore exclude list done send_file_list done send_files starting server_recv(2) starting pid=12320 recv_file_name(.) recv_file_name(maxim_file.dat) received 2 names recv_file_list done get_local_name count=2 /tmp/test_rsync/ recv_files(2) starting generator starting pid=12320 count=2 delta transmission enabled recv_generator(.,0) recv_generator(maxim_file.dat,1) generating and sending sums for 1 send_files(1, /tmp/test_rsync/maxim_file.dat) count=729 rem=128 blength=720 s2length=2 flength=524288 send_files mapped /tmp/test_rsync/maxim_file.dat of size 524288 calling match_sums /tmp/test_rsync/maxim_file.dat maxim_file.dat built hash table hash search b=720 len=524288 done hash search sending file_sum false_alarms=0 tag_hits=5722 matches=0 sender finished /tmp/test_rsync/maxim_file.dat generate_files phase=1 send_files phase=1 recv_files(maxim_file.dat) recv mapped maxim_file.dat of size 524288 got file_sum set modtime of .maxim_file.dat.HiyTJQ to (1195122277) Thu Nov 15 12:24:37 2007 renaming .maxim_file.dat.HiyTJQ to maxim_file.dat recv_files phase=1 generate_files phase=2 send files finished total: matches=0 tag_hits=5722 false_alarms=0 data=524288 recv_generator(.,0) set modtime of . to (1195122277) Thu Nov 15 12:24:37 2007 generate_files finished recv_files finished sent 524456 bytes received 4414 bytes 352580.00 bytes/sec total size is 524288 speedup is 0.99 _exit_cleanup(code=0, file=main.c, line=702): about to call exit(0) [EMAIL
Re: Problem with rsync recent file logic ?
Le jeudi 15 novembre 2007, Maxim Veksler a écrit : Hello, I have 2 servers I'm synchronizing using rsync, I have a situation where I : 1. rsync from rnd-dev2 to rnd-dev1 2. change the rsynched file on rnd-dev1 3. rsync from rnd-dev2 to rnd-dev1 again 4. File gets overridden on rnd-dev1 over though it has newer change time then file on rnd-dev2. I don't see any issue here. Note that file got rsynced from rnd-dev2 and had it's change date updated to older time (of rnd-dev2). As well as the content that got replaced by the out dated version on rnd-dev2. Bummer... What am I missing to get it do the right thing? I would expect rsync when called the second time from rnd-dev2 to recognize that the file on rnd-dev1 is newer and to NOT replaced it. Rsync has never check date recency by default, if time or size differ, rsync synchronize the file. Found in man page: -u, --updateskip files that are newer on the receiver which is probably what you need in this case. signature.asc Description: This is a digitally signed message part. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync error when sending lot's of small files
Hi All, I have a problem when transferring files from a rsyncd. This is my setup: * server: runs rsyncd on port 873. * firewall: forwarding for the 873 port from outside (using SNAT and DNAT). * client: manually running rsync command using modules. The problem occurs when lots of small identical files are being send. Small is smaller than 400 bytes. It also only occurs when the NATting happens, running it locally give no problems. Also when i use the --sockopts=SO_SNDBUF=xxx argument on the rsyncd it won't occur, but i noticed that the transfer is generally slower. (I also have normal sizes files available) I hope someone call help me or point me to the right direction. Greetings Johan Huysmans -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync error when sending lot's of small files
At 11:57 15.11.2007 +0100, you wrote: Hi All, I have a problem when transferring files from a rsyncd. This is my setup: What is the error you get? bye Fabi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync error when sending lot's of small files
whoops forgot to mention the error i receive ;) This message appears on the client side: rsync: writefd_unbuffered failed to write 4092 bytes [generator]: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(1122) [generator=2.6.9] rsync error: received SIGUSR1 (code 19) at main.c(1182) [receiver=2.6.9] Fabian Cenedese wrote: At 11:57 15.11.2007 +0100, you wrote: Hi All, I have a problem when transferring files from a rsyncd. This is my setup: What is the error you get? bye Fabi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--compare-dest not working for me. what am I doing wrong?
Hey. Can anybody spot my mistake? mkdir a b c echo foo a/test echo foo b/test rsync -av --compare-dest=b a/ c/ Expected results: no copy of a/test to c/test, since it is already present in b. Actual results: a/test gets copied to c/test What is wrong here? My expectation? my syntax? I'm having the same trouble with --link-dest, and --copy-dest. The behaviour I'm seeing from rsync, is what I would have expected if the --XX-dest= parameter would be missing. (I've tried every variation of having or not having slashes after the a/ b/ and c/'s I could think of) (tested with 2.6.9 on linux, and with cwrsync 2.6.9 on windows xp) Bye, Lucas -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync error?
Hi Folks Can anyone help me with the following error - is it serious; is the file transferred; are the other files transferred? [EMAIL PROTECTED] ~]$ rsync -a -X -A -z -v /home/morgan/Documents 192.168.1.30:/home/morgan/Documents [EMAIL PROTECTED]'s password: building file list ... done rsync: rsync_xal_set: lsetxattr(Documents/Projects/CBT/SH20/HearingSubmission/ManukauBridgeSubCBT_PostPatrick_3_2.doc,user.Beagle.AttrTime) failed: Permission denied (13) rsync: rsync_xal_set: lsetxattr(Documents/Projects/CBT/SH20/HearingSubmission/ManukauBridgeSubCBT_PostPatrick_3_2.doc,user.Beagle.Fingerprint) failed: Permission denied (13) rsync: rsync_xal_set: lsetxattr(Documents/Projects/CBT/SH20/HearingSubmission/ManukauBridgeSubCBT_PostPatrick_3_2.doc,user.Beagle.MTime) failed: Permission denied (13) rsync: rsync_xal_set: lsetxattr(Documents/Projects/CBT/SH20/HearingSubmission/ManukauBridgeSubCBT_PostPatrick_3_2.doc,user.Beagle.Uid) failed: Permission denied (13) sent 1572135 bytes received 20 bytes 89837.43 bytes/sec total size is 2063655599 speedup is 1312.63 rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9] [EMAIL PROTECTED] ~]$ Thanks, M. -- Getting errors: There are problems with the signature (or similar)? Update your system by installing certificates from CAcert Inc, see here: http://wiki.cacert.org/wiki/BrowserClients?#head-259758ec5ba51c5205cfb179cf60e0b54d9e378b Or, if Internet Explorer is your default browser, simply click this link: http://www.cacert.org/index.php?id=17 Morgan Read NEW ZEALAND mailto:mstuffATreadDOTorgDOTnz fedora: Freedom Forever! http://fedoraproject.org/wiki/Overview By choosing not to ship any proprietary or binary drivers, Fedora does differ from other distributions. ... Quote: Max Spevik http://interviews.slashdot.org/article.pl?sid=06/08/17/177220 RMS on fedora: http://fedoraproject.org/wiki/FreeSoftwareAnalysis/FSF smime.p7s Description: S/MIME cryptographic signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to make rsync faster?
I manage 250+ redhat linux boxes. The boxes are all setup the same way. On a daily basis, we sync the app directory which is about 30gb out to all hosts. The daily delta is actually less than 1gb, but since I can't be sure if any individual box was tempered during the day, I always do a full sync. On a monthly basis, we run with --delete to clean out the stale files on the hosts. The command I use daily is: /usr/bin/rsync -a -e ssh, with a ksh for loop on the 250+ host names The version is: rsync version 2.5.7 protocol version 26 Since rsync must do a chksum on the local and remote box on all files, the whole sync process takes over 2hrs even if nothing was changed. My questions are: 1) I know I have an old version, are there performance improvements in the later versions? I am not the SA, the process to request a new install is lengthy. 2) Is there a parallel rsync program? Looping 250 times to invoke causes rsync to checksum the local files 250 times, which is a waste of resource. Can parallel rsync be considered for a future version? 3) Are there better ways to achieve what I need to do with rsync or another tool? Thank you, Clayton - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice. IRS Circular 230 Disclosure: Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --compare-dest not working for me. what am I doing wrong?
On Thu, 2007-11-15 at 15:41 +0100, Lucas Meijer wrote: Can anybody spot my mistake? mkdir a b c echo foo a/test echo foo b/test rsync -av --compare-dest=b a/ c/ Relative --*-dest paths are interpreted relative to the destination directory (as stated in the man page), so you should use: rsync -av --compare-dest=../b a/ c/ Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync error?
On Nov 15, 2007, at 10:55 AM, Morgan Read wrote: Can anyone help me with the following error - is it serious; is the file transferred; are the other files transferred? The file is transferred as are the other files. However, rsync probably gives a non-zero exit status. lsetxattr(Documents/Projects/CBT/SH20/HearingSubmission/ ManukauBridgeSubCBT_PostPatrick_3_2.doc,user.Beagle.AttrTime) failed: Permission denied (13) The things that failed to copy are meta-data used by beagle (the gnome desktop search tool). This meta-data is expendable. Don't worry. As a point of interest; what version of rsync is this, and was the file read-only? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Filenames
Hello, there are problems syncing files with special charcters in the filename. The special characters in the filename are displayed as a question mark. During synchronization, the rsync tells the file is vanished. Here is the standard output with level 1 verbosity. Higher verbosity levels dont give more informations about the problem. The stat command with this file succeed. file has vanished: /backupmnt/neef/backup2/Sch?newolf.JPG sent 60557 bytes received 64 bytes 121242.00 bytes/sec total size is 188702060 speedup is 3112.82 rsync warning: some files vanished before they could be transferred (code 24) at main.c(977) [sender=2.6.9] Is rsync able to sync files with this file? Help would be appreciated, thanks in advance, Benjamin Neef. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to make rsync faster?
On Nov 15, 2007 9:08 PM, Tang, Clayton (Yiqi) [EMAIL PROTECTED] wrote: I manage 250+ redhat linux boxes. The boxes are all setup the same way. On a daily basis, we sync the app directory which is about 30gb out to all hosts. The daily delta is actually less than 1gb, but since I can't be sure if any individual box was tempered during the day, I always do a full sync. On a monthly basis, we run with --delete to clean out the stale files on the hosts. The command I use daily is: /usr/bin/rsync -a -e ssh, with a ksh for loop on the 250+ host names The version is: rsync version 2.5.7 protocol version 26 Since rsync must do a chksum on the local and remote box on all files, the whole sync process takes over 2hrs even if nothing was changed. My questions are: 1) I know I have an old version, are there performance improvements in the later versions? I am not the SA, the process to request a new install is lengthy. 2) Is there a parallel rsync program? Looping 250 times to invoke causes rsync to checksum the local files 250 times, which is a waste of resource. Can parallel rsync be considered for a future version? 3) Are there better ways to achieve what I need to do with rsync or another tool? Thank you, Clayton Hello Tang, First, for such operation you should RTFM about rsync batch mode [1]. Second, If I were you I would look for other solutions. Perhaps a shared NFS storage or a copied FS based on drbd. Using rsync sounds like a quick hack to me when you had 2 servers and 0 time to market. I would love to hear other suggestions people have on this list for your issue. [1] http://samba.anu.edu.au/ftp/rsync/rsync.html -- Cheers, Maxim Veksler Free as in Freedom - Do u GNU ? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: 2.6.9 w/ acl, xattr, and fake-super support?
On Nov 15, 2007, at 3:02 PM, Daniel Maher wrote: As i cannot get 3.0.0pre5 to work in my environment (throwing crazy errors which i've posted previously), i would like to revert to 2.6.9 . Of course, the reason i tried to use v3 in the first place was for the acl, xattr, and fake-super options - which, evidently, can be enabled under 2.6.9, as per the following sources: http://lists.samba.org/archive/rsync/2007-February/017218.html http://fresh.t-systems-sfr.com/unix/src/misc/rsync-2.6.9.tar.gz:a/ rsync-2.6.9/patches/fake-super.diff I can successfully apply both acls.diff and xattrs.diff to the 2.6.9 source, however, fake-super.diff is not as smooth: $ patch -p1 patches/fake-super.diff Did you apply first acfs, then xattrs, then fake-super? You need to apply them in that order. If i attempt to compile, make completes successfully (albeit with some warnings), and i get an rsync binary. ... which clearly will be broken. I am curious if anybody has successfully enabled acl, xattr, and fake-super support in 2.6.9 . If so, i would /love/ to know how you did it, as i'm starting to get a little desperate for a solution! I had no problems when I did this. Can you give the complete sequence of commands you are running, from untarring to patching the fake-super patch? (again: first acl, then xattr) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync error?
On Thu, 2007-11-15 at 19:37 +0100, Wesley W. Terpstra wrote: On Nov 15, 2007, at 10:55 AM, Morgan Read wrote: Can anyone help me with the following error - is it serious; is the file transferred; are the other files transferred? The file is transferred as are the other files. However, rsync probably gives a non-zero exit status. lsetxattr(Documents/Projects/CBT/SH20/HearingSubmission/ ManukauBridgeSubCBT_PostPatrick_3_2.doc,user.Beagle.AttrTime) failed: Permission denied (13) The things that failed to copy are meta-data used by beagle (the gnome desktop search tool). This meta-data is expendable. Don't worry. As a point of interest; what version of rsync is this, and was the file read-only? Thanks Wesley Yes, it was read only file - but, only one of many in that particular directory. Version of rsync: [EMAIL PROTECTED] ~]$ rpm -q rsync rsync-2.6.9-3.fc7 Regards, M. -- Getting errors: There are problems with the signature (or similar)? Update your system by installing certificates from CAcert Inc, see here: http://wiki.cacert.org/wiki/BrowserClients?#head-259758ec5ba51c5205cfb179cf60e0b54d9e378b Or, if Internet Explorer is your default browser, simply click this link: http://www.cacert.org/index.php?id=17 Morgan Read NEW ZEALAND mailto:mstuffATreadDOTorgDOTnz fedora: Freedom Forever! http://fedoraproject.org/wiki/Overview By choosing not to ship any proprietary or binary drivers, Fedora does differ from other distributions. ... Quote: Max Spevik http://interviews.slashdot.org/article.pl?sid=06/08/17/177220 RMS on fedora: http://fedoraproject.org/wiki/FreeSoftwareAnalysis/FSF smime.p7s Description: S/MIME cryptographic signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync error?
On Thu, 2007-11-15 at 11:31 +0100, Daniel Maher wrote: On Thu, 2007-11-15 at 22:55 +1300, Morgan Read wrote: [EMAIL PROTECTED] ~]$ rsync -a -X -A -z -v /home/morgan/Documents 192.168.1.30:/home/morgan/Documents [EMAIL PROTECTED]'s password: building file list ... done rsync: rsync_xal_set: lsetxattr(Documents/Projects/CBT/SH20/HearingSubmission/ManukauBridgeSubCBT_PostPatrick_3_2.doc,user.Beagle.AttrTime) failed: Permission denied (13) rsync: rsync_xal_set: ... Can user morgan set ACLs on the filesystem of the receiving machine? If so, can user morgan set ACLs to the exactly the same state as the sending machine? If the answer to either of these is no, then you're going to run into problems like the one you've described. Don't know but suspect no is the answer to both Qs: destination system is my old laptop running fc5 that I'm begining to use as a mirror, with a file system that pre-dates fc5 (perhaps 4 or 3). Not sure ACLs were up and running on it. Once everything is sorted Ill be updating it. Regards, M. -- Getting errors: There are problems with the signature (or similar)? Update your system by installing certificates from CAcert Inc, see here: http://wiki.cacert.org/wiki/BrowserClients?#head-259758ec5ba51c5205cfb179cf60e0b54d9e378b Or, if Internet Explorer is your default browser, simply click this link: http://www.cacert.org/index.php?id=17 Morgan Read NEW ZEALAND mailto:mstuffATreadDOTorgDOTnz fedora: Freedom Forever! http://fedoraproject.org/wiki/Overview By choosing not to ship any proprietary or binary drivers, Fedora does differ from other distributions. ... Quote: Max Spevik http://interviews.slashdot.org/article.pl?sid=06/08/17/177220 RMS on fedora: http://fedoraproject.org/wiki/FreeSoftwareAnalysis/FSF smime.p7s Description: S/MIME cryptographic signature -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Problem with --exclude command
Hi everybody, i want to exclude a set of directories of a rsync synchronization but i have a problem : --- serv-dev:~/scripts_Admin# RSYNC_PASSWORD= /usr/bin/rsync -vv --checksum --recursive --links --exclude .svn/ --exclude test/ /var/www/ssl/ [EMAIL PROTECTED]::ssl/ opening tcp connection to 192.168.130.1 port 873 opening connection using --server -vvlrc . ssl/ building file list ... [sender] excluding directory .svn because of pattern .svn/ [sender] excluding directory test because of pattern test/ [sender] excluding directory downloads/test because of pattern test/ done total: matches=0 tag_hits=0 false_alarms=0 data=0 sent 1166 bytes received 16 bytes 472.80 bytes/sec total size is 66022402 speedup is 55856.52 --- So downloads/test is excluded from the rsync but i just want my ./test/ folder being excluded from the transfer, not my downloads/test/ folder. I've simplified but all my exclusions are generated from a command like `ls -F | grep -v downloads/` so it should be compatible with the syntax used by ls output. I've search on rsync man but i'm a little bit lost/ Thanks very much for your help. Alex -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to make rsync faster?
1) Yes! 2.6.x especially helps with memory. 2) Not that I've seen, but I'd be really interested! 3) We've had great luck with (Open)AFS, though it's not for everyone, not even in our environment. =) (having to load a kernel module being #1 complaint). rsync allows us to accommodate those that don't wish to use AFS. Here's what we're doing: Roughly 10% (~1k hosts) of our install-base use rsync as an alternative to AFS (our system configuration and application store). About 250M is checked hourly, though as often as every 15 minutes for more time sensitive systems. We've tossed around the idea of using batch-mode, but it unfortunately doesn't fit our model - It's basically a huge buffet of data that the hosts pick and choose which trees to keep in sync. What we've found is client initiated pulls scale much better than pushes from a central server. We have each host sleep for a random amount of time using the hostname as a seed (so it's the same from run to run) before initiating the rsync. This causes multiple rsyncs to be run on the server, but it can handle dozens of connections at a time without issue, especially after the switch to 2.6 versions of rsync. We also have multiple servers from which the client can rsync from, but that is handled similarly to the timing: A host randomly picks a server from a list using hostname as the seed. The servers are monitored for load and new ones added appropriately. Our server to client ratio is close to 50:1. -Ducky Tang, Clayton (Yiqi) wrote: I manage 250+ redhat linux boxes. The boxes are all setup the same way. On a daily basis, we sync the app directory which is about 30gb out to all hosts. The daily delta is actually less than 1gb, but since I can't be sure if any individual box was tempered during the day, I always do a full sync. On a monthly basis, we run with --delete to clean out the stale files on the hosts. The command I use daily is: /usr/bin/rsync -a -e ssh, with a ksh for loop on the 250+ host names The version is: rsync version 2.5.7 protocol version 26 Since rsync must do a chksum on the local and remote box on all files, the whole sync process takes over 2hrs even if nothing was changed. My questions are: 1) I know I have an old version, are there performance improvements in the later versions? I am not the SA, the process to request a new install is lengthy. 2) Is there a parallel rsync program? Looping 250 times to invoke causes rsync to checksum the local files 250 times, which is a waste of resource. Can parallel rsync be considered for a future version? 3) Are there better ways to achieve what I need to do with rsync or another tool? Thank you, Clayton - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice. IRS Circular 230 Disclosure: Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --compare-dest not working for me. what am I doing wrong?
Hey Matt, Relative --*-dest paths are interpreted relative to the destination directory (as stated in the man page), so you should use: rsync -av --compare-dest=../b a/ c/ tried that too, same result for me: mkdir a b c echo foo a/test echo foo b/test rsync -av --compare-dest=../b a/ c/ expected: no copy actual result: a/test gets copied to c/test (also tested with --compare-dest=../b/ --compare-test=/b/, and the other things I actually tried I'm to emberrassed to write here :) ) Bye, Lucas -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How to make rsync faster?
I'm no rsync guru my any means, but two things spring to mind. Use the -t option to stop all the spurious check summing. Split your script into multiple scripts, each with a share of host names. Run each in parallel. Multiple rsyncs can run on the one box concurrently. Craig -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tang, Clayton (Yiqi) Sent: Friday, 16 November 2007 6:09 AM To: rsync@lists.samba.org Subject: How to make rsync faster? I manage 250+ redhat linux boxes. The boxes are all setup the same way. On a daily basis, we sync the app directory which is about 30gb out to all hosts. The daily delta is actually less than 1gb, but since I can't be sure if any individual box was tempered during the day, I always do a full sync. On a monthly basis, we run with --delete to clean out the stale files on the hosts. The command I use daily is: /usr/bin/rsync -a -e ssh, with a ksh for loop on the 250+ host names The version is: rsync version 2.5.7 protocol version 26 Since rsync must do a chksum on the local and remote box on all files, the whole sync process takes over 2hrs even if nothing was changed. My questions are: 1) I know I have an old version, are there performance improvements in the later versions? I am not the SA, the process to request a new install is lengthy. 2) Is there a parallel rsync program? Looping 250 times to invoke causes rsync to checksum the local files 250 times, which is a waste of resource. Can parallel rsync be considered for a future version? 3) Are there better ways to achieve what I need to do with rsync or another tool? Thank you, Clayton - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice. IRS Circular 230 Disclosure: Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --compare-dest not working for me. what am I doing wrong?
On Thu, Nov 15, 2007 at 10:29:58PM +0100, Lucas Meijer wrote: mkdir a b c echo foo a/test echo foo b/test rsync -av --compare-dest=../b a/ c/ The two test files don't match in their preserved attributes. They must either have the same mtime, or you must not ask rsync to preserve times. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
