RE: question on switches and their order
Jeff Ordering of switches is important, but only for the --include* --exclude* switches. For your sample command there should be no difference. We also use NetApp filers and exclude .snapshot, the same way you have. It works fine... The --delete command won't help here as it will delete everything on the destination that is not on the source but will not delete .snapshot. To do that you need --delete-excluded. However I don't think that is your solution. Using --exclude is what you want, hopefully you can find a typo somewhere that is causing you grief. With compression you will end up doing the compression and decompression on the same box -it slows things down big time. Effectively, as far as your machine is concerned, both directories are on the same box. Even if you were using two different hosts on a gb lan, we have found that compression slows things down by about 60%. The CPU's simply cannot keep up with the LAN. The other switch you should use is -W. It deals with whole files and is significantly quicker. Cheers Mark -Original Message- From: Jeff Kennedy [mailto:[EMAIL PROTECTED]] Sent: Tuesday, 26 June 2001 08:15 To: Rsync List Subject: question on switches and their order Does it matter what order the options/switches are placed? I ask because I did an initial rsync of data from a NetApp to a SAN box using the following: rsync -avz --exclude=.snapshot /mail-data/ /mnt The .snapshot directory is only useful to Ontap in this instance so I did not want it transfered. It seemd to work correctly and all was fine. Until I tried to run an update using the following: rsync -avz --exclude=.snapshot --delete /mail-data/ /mnt This transfered the .snapshot directory and filled the partition. Is the above incorrect? Should it be like this: rsync --exclude=.snapshot --delete -avz /mail-data/ /mnt Also, do I need the 'z' if I have a gigabit network? I think it would put more strain on the filer than necessary and actually slow transfer time. Thanks. -- = Jeff Kennedy Unix Administrator AMCC [EMAIL PROTECTED] CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
RE: question on switches and their order
Leave out the --delete. It will delete everything on the destination that is not present on the source, except for .snapshot -if present. If you want to play with --delete use the -n option as well. This will tell you what rsync is going to do without actually doing anything. Suggest you try rsync -avW --exclude=.snapshot /mail-data/ /mnt Cheers Mark -Original Message- From: Jeff Kennedy [mailto:[EMAIL PROTECTED]] Sent: Tuesday, 26 June 2001 09:19 To: RSync List (E-mail) Subject: Re: question on switches and their order So, if I understand you correctly, it should work fine (with improvements) like this: rsync --exclude=.snapshot --delete -avW /mail-data/ /mnt Is this correct? Thanks. CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
RE: librsync and network functions
Martin To tridge and I, rsync 2.x is essentially in maintenance mode at the moment: it doesn't seem like a good investment to add more features or functionality to the codebase. A better way to improve it is to build a new, cleaner protocol. Does this mean there is a version 3.x on its way? If so when is the likely release date? Cheers Mark CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
RE: rsync hangs in select on linux 2.4.5
Wayne Umm, errr, I have to confess that I don't know how to apply your hang patch... I did try the patch command, but to no avail. A quick explanation would be appreciated. I have: -downloaded the rsync-2.4.6 source and unpacked it. -am using Solaris 2.6. -am using Solaris compilers and tools. Also the patch you list below (link) is different to one I saved of one of your earlier emails. Is the one below the latest version of the anti hang patch? Cheers Mark -Original Message- From: Wayne Davison [mailto:[EMAIL PROTECTED]] Sent: Thursday, 14 June 2001 14:02 To: Jurgen Botz Cc: [EMAIL PROTECTED] Subject: Re: rsync hangs in select on linux 2.4.5 On Wed, 13 Jun 2001, Jurgen Botz wrote: I'm seeing the following bizarre behavior... - rsync -av from one local fs to another local fs (local disk on both) - rsync gets to very near completion then hangs indefinitely - attach strace to rsync process doing the 'copy from' and it starts up again and finishes! This is the second of two hangs that my recent nohang patch fixes. The patch is relative to the CVS version, but it applies fine to version 2.4.6 as well (with offsets, but no fuzz). I have not looked at 2.4.5, though. You can get the patch from here: http://www.clari.net/~wayne/rsync-nohang.patch ..wayne.. CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
RE: rsync fails
Rsh only sets up a limited path for you. Do the command: rsh remote_host echo \$PATH (or ssh remote_host echo \$PATH) and this will tell you what your remote PATH is. Perhaps install rsync into one of these directories or use sym links. If you are using ssh have a look at the man page about changing the path. Personally I solved this by creating a script in a dir on my local path, say /usr/local/bin, called rsync that has a line in it similar to the one below: /opt/rsync/bin/rsync --rsync-path=/opt/rsync/bin/rsync $* Mark -Original Message- From: David Salisbury [mailto:[EMAIL PROTECTED]] Sent: Friday, 18 May 2001 11:09 To: [EMAIL PROTECTED]; Bob Foster Subject: Re: rsync fails I just got done dealing with this problem. It was solved for me using the --rsync-path=/usr/local/bin/rsync flag, specifying the remote location of rsync. It be great if someone knew how to avoid using this parameter. cheers, Dave CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
RE: FW: Problem with large include files
How do I go about registering this bug with the include file. It would be good get this bug fixed as I would like to be able to back to 2.4.6 (or whatever) as it is faster and it has bandwidth limiting. Will let you know the results of the testing. Cheers Mark -Original Message- From: Dave Dykstra [mailto:[EMAIL PROTECTED]] Sent: Wednesday, 16 May 2001 01:24 To: Wilson, Mark - MST Cc: RSync List (E-mail) Subject: Re: FW: Problem with large include files On Tue, May 15, 2001 at 03:31:23PM +1200, Wilson, Mark - MST wrote: ... Do you have any idea what the maximum number of files you can have in an include file is (for the current version)? No, I don't. It probably depends on a lot of variables. How do you want your test on 2.3.2 done? ie LAN or high speed WAN, numbers of file, sizes of files, things to time, daemon v rsh. What I'd like to see is a case that might make the biggest difference with and without the optimization: - probably use the LAN - the largest number of files that you can get to work - small files - time the whole run with the time command, CPU time and elapsed time - I don't know about daemon vs rsh, but the daemon leaves the most under rsync's control so that may be preferable - Dave Dykstra CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
FW: Problem with large include files
Try rsync 2.3.2 on both ends and see what the result is. What transport method are you using (rsh, ssh, or rsync --daemon mode)? --daemon mode. Some tests a colleague did showed a performance advantage. Come to think of it, I know from experience that rsync is able to keep the TCP queues full in both directions during a transfer. There must be something else going on. Does your link perhaps lose a percentage of the packets, forcing TCP to timeout and retransmit, slowing it down? I thought rsync *should* be quite good, given what it is used for. The network guys tell me that the link only sheds packets (ATM frames) in an intelligent way when the bandwidth goes over 29.9 Mbit. If only a single stream is used no packets are lost. Up until I started using parallel rsync streams, no one had been able to soak the link. I have a theory that TCP requires some kind of ACK or NAK for every packet sent. Or if it uses sliding windows the window size is too small for the speed/latency of this link, hence TCP doesn't send flat out. Unfortunately I've never delved into TCP enough to find out how this part of it works. Is there are version of rsync that uses UDP? If implemented correctly this would get around any latency problems. Cheers Mark CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
RE: Problem with large include files
Dave A couple of points: 1. I think you are telling me that if I go back to 2.3.2 my problems should go away. Is this correct? 2. I rather oversimplified how I am using rsync... Perhaps I had better explain. If I can do some testing for you I am happy to do so, however there is quite a bit of pressure for me to get my problem fixed. This I must do first. Environment: I am shifting large applications from a Sun E10k domain backended with a NetApp F840 Filer to another Sun E10k backended with a NetApp F840 Filer. The source machine is in Auckland, New Zealand and the destination machine is in Melbourne, Australia. The link is a 10Mbit pipe that is burstable to 30Mbit (actually I can run it full time at 30Mbit at the moment, I'm sure that wont last). The latency on the link is about 20ms one way. Original Problem: Because of the slow link and requirement to maintain all the files, links, etc we were limited to a few mechanisms. Originally we were ftping tgz files but finding sufficient scratch space was a problem. Also the compress time made things slow. We also tried various versions of rsh tar gzip, etc. Eventually I tried rsync because it did on the fly compression and correctly handled permissions, files, links, etc. Locally, especially across our 1Gbit links, rsync flew. Everyone was very happy. However it wasn't so fast across the Tasman (sea between NZ and Oz). In fact with a bit of sniffing by the network guys we found that rsync wasn't even using 1Mbit of the link. H, latency methinks. So I wrote a fancy perl script that took the required directories to be transferred and split all the files into 20 (configurable) balanced streams. This meant 20 copies of rsync running at the same time. Of course the balancing was done on the uncompressed file sizes so took no account of how dense the files were. This is unfortunate as different streams take different amounts of times. However the results were quite spectacular! At best, on DB files, we were getting disk to disk transfer rates of up to 1.5Gbyte per minute. Not bad over a 10Mbit link... It completely soaked the link and maxed out the 12 processors. Fun having all that power at your command! The current problem: However the euphoria rapidly wore off when I tried to transfer 128Gb of test data, some of it quite dense with many, many small files. In fact 104755 files in total. Unfortunately if there are too many files in an include file (--include-from) the streams quit part way through. Very upsetting... I tried turning up the debugging and re running it. I didn't find out anything except how many files it did before it quit. Interestingly enough, with heaps of debug, it processed more files... Weird. Anyway, the purpose of all this verbosity is two fold. Firstly you need to tell me, given my environment, how you want your testing done and secondly if you had any ideas of how to fix my problem. If we can't fix it we will have to do the backup to tape and send it on a plane method -which we really want to avoid. As a thought, have you or any of the other developers thought of getting rsync to operate over a number of streams or to use sliding windows to overcome latency effects? I look forward to your reply. Cheers Mark -Original Message- From: Dave Dykstra [mailto:[EMAIL PROTECTED]] Sent: Saturday, 12 May 2001 01:43 To: Wilson, Mark - MST Cc: RSync List (E-mail) Subject: Re: Problem with large include files On Fri, May 11, 2001 at 11:41:41AM +1200, Wilson, Mark - MST wrote: Hi there I recently tried to do a transfer of a directory tree with about 120,000 files. I needed to select various files and used the --include-from option to select the files I needed to transfer. The include file had 103,555 filenames in it. The problem I have is that the transfer quit after transferring some of the files. I am running the remote end in daemon mode. Interestingly the remote daemon spawned for this job was left behind and did not quit. I had to kill it when I noticed it many hours later. Unfortunately I didn't have any -v options so didn't get any information as to what caused it. I will be doing further tests to see if I can get more information. Are there any restrictions on the amount of files you can have in an include file? The two machines are Sun E1 domains with 12 processors and 12288 Megabytes of RAM. Any ideas on how to crack this would be appreciated. Ah, perhaps we finally have somebody to perform the test I have been asking for for months. Some background: prior to rsync version 2.4.0 there was an optimization in rsync, which I put in back when I was officially maintaining rsync, that would kick in whenever there was a list of non-wildcard include patterns followed by an exclude '*' (and when not using --delete). The optimization bypassed the normal recursive traversal of all the files and directly opened the included files and sent the list over. A side effect was that it did
Problem with large include files
Hi there I recently tried to do a transfer of a directory tree with about 120,000 files. I needed to select various files and used the --include-from option to select the files I needed to transfer. The include file had 103,555 filenames in it. The problem I have is that the transfer quit after transferring some of the files. I am running the remote end in daemon mode. Interestingly the remote daemon spawned for this job was left behind and did not quit. I had to kill it when I noticed it many hours later. Unfortunately I didn't have any -v options so didn't get any information as to what caused it. I will be doing further tests to see if I can get more information. Are there any restrictions on the amount of files you can have in an include file? The two machines are Sun E1 domains with 12 processors and 12288 Megabytes of RAM. Any ideas on how to crack this would be appreciated. Cheers Mark Mark Wilson Unix Consultant Midrange Services Team Corporate Information Technology Air New Zealand Phone +64-9 306-5480 x65-480 Fax +64-9 306-5457 Email: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
Sending large file across a 10Mbit link
Hi there I am new to rsync lists so may be posting this message to the wrong place. Please let me know if I am. We are using rsync to to send large files (larger than 2Gb) across a 10Mbit link (burstable to 30Mbit) using rsync. The link has a return latency of about 52ms. In speed comparisons ftp is transferring a 2G file in 8 minutes while rsync is taking 21 minutes. The file being sent is compressed and doesn't exist at the destination at time of sending. Compression option is not used in rsync. I was wondering about the possibility of using the socket options in rsyncd.conf to tune the transfer for greater speed but cannot find any information to help with this. Does anyone have any ideas. Cheers Mark Mark Wilson Unix Consultant Midrange Services Team Corporate Information Technology Air New Zealand Phone +64-9 306-5480 x65-480 Fax +64-9 306-5457 Email: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
RE: Sending large file across a 10Mbit link
Hmm, under those conditions, to be honest, is rsync even appropriate? You could just copy the file with rcp/scp (heck, or even ftp). You're not benefitting at all from rsync's algorithm. Oops, there was a mistake. The compressed file was 227Mb (2G uncompressed). The features we want to use in rsync are 1. Gets links, permissions, devices, etc correct. 2. Update in some cases. 3. compression on the fly. Apart from the update the other requirements can be met by other methods such as rsh "tar | gzip" etc. However the rsh - tar method is considerably slower. As you say our pipe is relatively fast with low latency, however compared with our LAN (100Mb and 1Gb) it is slow and has large latency. We have a huge amount of data to move so getting a fast method is important. FTP is way faster in the transfer but suffers from us having to "prepare" the files before sending (tar gzip) which takes time and requires disk space, adding up to more bother and no real time saving overall. Rsync is easy to use and does everything we need. Unfortunately the transfer is much slower, hence the desire to attempt to tune it. I had wondered about increasing the transfer/receive buffers via the socket options, but don't know what the current sizes are. Assuming rsync though, a better comparison to your ftp would be to use he -W option of rsync, which will just copy the file entirely and not I'll give it a bash. Would that make much difference for one file? Cheers Mark CAUTION - This message may contain privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message you are hereby notified that any use, dissemination, distribution or reproduction of this message is prohibited. If you have received this message in error please notify Air New Zealand immediately. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of Air New Zealand. _ For more information on the Air New Zealand Group, visit us online at http://www.airnewzealand.com or http://www.ansett.com.au _
