RE: How to make rsync faster?
-t is good enough for me. It checks size and time and would be heaps faster than chksum. It has never caused me a problem as yet. Whether size time over chksum is good enough for you is up to you. Craig... -Original Message- From: Tang, Clayton (Yiqi) [mailto:[EMAIL PROTECTED] Sent: Saturday, 17 November 2007 1:16 AM To: Craig Hammond; rsync@lists.samba.org Subject: RE: How to make rsync faster? Thanks for the reply. How safe is it NOT to chksum? Does rsync use size instead or date instead or both together? Actually splitting is what I just did. I split 250 hosts into 4 lists and running 4 rsync jobs from the master in parallel. This causes 80%-90% total CPU usage, still runs for about 50min... Regards, Clayton -- Clayton (Yiqi) Tang, LMX / Autotrader Production Management 212-526-7493, 745-7th Ave, New York, NY 10019 -Original Message- From: Craig Hammond [mailto:[EMAIL PROTECTED] Sent: Thursday, November 15, 2007 5:57 PM To: Tang, Clayton (Yiqi); rsync@lists.samba.org Subject: RE: How to make rsync faster? I'm no rsync guru my any means, but two things spring to mind. Use the -t option to stop all the spurious check summing. Split your script into multiple scripts, each with a share of host names. Run each in parallel. Multiple rsyncs can run on the one box concurrently. Craig -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tang, Clayton (Yiqi) Sent: Friday, 16 November 2007 6:09 AM To: rsync@lists.samba.org Subject: How to make rsync faster? I manage 250+ redhat linux boxes. The boxes are all setup the same way. On a daily basis, we sync the app directory which is about 30gb out to all hosts. The daily delta is actually less than 1gb, but since I can't be sure if any individual box was tempered during the day, I always do a full sync. On a monthly basis, we run with --delete to clean out the stale files on the hosts. The command I use daily is: /usr/bin/rsync -a -e ssh, with a ksh for loop on the 250+ host names The version is: rsync version 2.5.7 protocol version 26 Since rsync must do a chksum on the local and remote box on all files, the whole sync process takes over 2hrs even if nothing was changed. My questions are: 1) I know I have an old version, are there performance improvements in the later versions? I am not the SA, the process to request a new install is lengthy. 2) Is there a parallel rsync program? Looping 250 times to invoke causes rsync to checksum the local files 250 times, which is a waste of resource. Can parallel rsync be considered for a future version? 3) Are there better ways to achieve what I need to do with rsync or another tool? Thank you, Clayton - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice. IRS Circular 230 Disclosure: Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without
RE: How to make rsync faster?
I'm no rsync guru my any means, but two things spring to mind. Use the -t option to stop all the spurious check summing. Split your script into multiple scripts, each with a share of host names. Run each in parallel. Multiple rsyncs can run on the one box concurrently. Craig -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tang, Clayton (Yiqi) Sent: Friday, 16 November 2007 6:09 AM To: rsync@lists.samba.org Subject: How to make rsync faster? I manage 250+ redhat linux boxes. The boxes are all setup the same way. On a daily basis, we sync the app directory which is about 30gb out to all hosts. The daily delta is actually less than 1gb, but since I can't be sure if any individual box was tempered during the day, I always do a full sync. On a monthly basis, we run with --delete to clean out the stale files on the hosts. The command I use daily is: /usr/bin/rsync -a -e ssh, with a ksh for loop on the 250+ host names The version is: rsync version 2.5.7 protocol version 26 Since rsync must do a chksum on the local and remote box on all files, the whole sync process takes over 2hrs even if nothing was changed. My questions are: 1) I know I have an old version, are there performance improvements in the later versions? I am not the SA, the process to request a new install is lengthy. 2) Is there a parallel rsync program? Looping 250 times to invoke causes rsync to checksum the local files 250 times, which is a waste of resource. Can parallel rsync be considered for a future version? 3) Are there better ways to achieve what I need to do with rsync or another tool? Thank you, Clayton - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice. IRS Circular 230 Disclosure: Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
differential backup using deltas.
I was wondering if the following is possible, or is planned for rsync. I backup a number of databases with rsync that work very well with the rsync algorithm. Ie, I am backing up a gigabyte+ database over a dial-up speed link and it doesn't take very long, so the daily differences in the files are obviously very small compared to it's total size. I am storing many copies of this database as a backup at the moment, which take up a GB every backup. It would be a lot more efficient on disk space if I only had to store the original, then the rsync differences. Then have some way way to combine the two to end up with the complete modified file. Craig.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync problems
This probably won't help, but I had a similar problem with rsync 2.6.9. Cwrsync with rsync 2.6.9 on Windows 2003 server to OpenBSD. (rsync 2.6.8, not using daemon mode) 2.6.8 at both ends worked perfectly. Upgraded to 2.6.9 just on the Windows box and it would just hang at a diffent spot each time. Downgraded the Windows box to 2.6.8 and it works fine again. I assumed it was something I was doing wrong, so once I got it working on 2.6.8 I left it alone. I'm only mentioning it now as I saw someone else with a similar problem with 2.6.9. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Wayne Davison Sent: Thursday, 16 November 2006 11:57 AM To: William McInnis Cc: rsync@lists.samba.org Subject: Re: rsync problems Did you follow my last instructions? It sounds like you're talking to an rsync daemon, so you'll need to make sure that the client is given the same timeout option on the command-line that the daemon has configured into it so that it can send periodic keep-alive messages. Even, then, if your timeout is too short, nothing will avoid some timeouts except for making it longer. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsyncing from Windows 2003 shadow copies.
Hi, Hopefully this isn't duplicating info. If so, sorry for the noise. I tried searching through old posts to this list about this issue, but couldn't find anything that worked. Below is a script I have put together that lets you rsync from a Windows 2003 shadow copy without having to copy files out of it first. You can link the shadow copy to either a drive letter or to a folder. You then rsync from there. The shadow copy is then deleted when the script finishes. Hopefully this helps someone. you need vshadow.exe which is part of the VSS SDK which is a free download from MS. http://www.microsoft.com/downloads/details.aspx?FamilyID=0B4F56E4-0CCC-4 626-826A-ED2C4C95C871displaylang=en I'm sure this can be improved on. I welcome any feedback. SHADOWRSYNC.CMD setlocal @REM test if we are called by shadowrsync.cmd if NOT %CALLBACK_SCRIPT%== goto :IS_CALLBACK @REM @REM Set these variables to what you need. @REM SHADOW_DRIVE_LETTER is an existing drive you want to shadow @REM @REM MAP_SHADOW_TO needs to be either a drive letter that isn't @REM currently in use, or an empty directory @REM eg. x: or c:\temp\shadowcopy @REM set SHADOW_DRIVE_LETTER=D: set MAP_SHADOW_TO=X: @REM @REM Create the shadow copy - and generate env variables into a temporary script. @REM Then resursively come back into this script. set CALLBACK_SCRIPT=%~dpnx0 set TEMP_GENERATED_SCRIPT=GeneratedTempScript.cmd @echo ...Create the shadow copy... vshadow.exe -nw -p -script=%TEMP_GENERATED_SCRIPT% -exec=%CALLBACK_SCRIPT% %SHADOW_DRIVE_LETTER% del /f %TEMP_GENERATED_SCRIPT% @goto :EOF :IS_CALLBACK setlocal @REM Get the vshadow temp variables. call %TEMP_GENERATED_SCRIPT% @REM Map the drive to the shadow copy vshadow -el=%SHADOW_ID_1%,%MAP_SHADOW_TO% @REM Now do your rsync thing rsync blah blah blah blah @REM Delete the shadow vshadow -ds=%SHADOW_ID_1% -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Backup Exchange mailboxes using rsync
I do an offline copy of the exchange database, but (as you indicated) as it can be tricky to use on another server, as seconday backup I use EXMERGE to extract everyones mailbox to a PST file. I then rsync the PST files. It has some limitations, but it works ok as a secondary backup. I've used EXMERGE on Exchange 2000 2003. Not sure if it works with 5.5, but it might. Craig. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Reuben Pearse Sent: Thursday, 3 August 2006 7:43 PM To: rsync@lists.samba.org Subject: Backup Exchange mailboxes using rsync Hi all, Has anyone got any advice on the best way to backup Exchange mailboxes (Exchange 5.5) running on a Windows 2000 server? We already use rsync on a number of other Windows boxes for general file backup. I've read that even if you get backups of the Exchange mailbox files, then it's really tricky to get the mailboxes back into Exchange if the server dies. Anyone got any tips on this? Thanks in advance Reuben [EMAIL PROTECTED] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
changing bwlimit at runtime.
I am using rsync in a situation where the amount of bwlimit I want to use changes depending on the time of day. for example, I startrsync at 2am when no-one else is using the system, so I want it to use100k/sec. At 9am, it is still running, but as the bandwidth of the link it's using now needs to be shared with other users, I would likersync to only use 10k/sec. Any way of doing this without stoppingand restarting it. Thanks, Craig. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
exclude an excluded file from being deleted by --delete-excluded
Hi, I have a particular directory in my exclude list, but I want it excluded from --delete-excluded. Is this possible. or maybe if I explain what I'm trying to do, someone will come up with a more suitable way. I am using rsync (2.6.6) to backup data from a Windows Server (cygwin) to an OpenBSD box. rsync -rtvz --delete-excluded --del --timeout=3600 -e ssh -i c:\cygwin\ssh-key --exclude-from=/cygdrive/d/rsync/rsync-d-exclude.txt --bwlimit=5 /cygdrive/d [EMAIL PROTECTED]:/var/rsync/user/backup/server So basically, I'm backing up the whole D: drive of the windows server except for what I exclude. I run this at about 6pm to capture all the data from the days work. One of the directories I exclude has a database in it as it is always still in use at this time. At about 2am, I run another instance of rsync to copy just the directory containing the database. My problem is that as the database directory is in the exclude list, it gets deleted because of the --delete-excluded. I do however want the --delete-excluded active for the remainder of the directories. Thanks, Craig. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
