Re: sync performance falls off a cliff
Mike Connell wrote: Hi, Hi Mike, I've got identical servers. One is primary the other is backup receiving rsyncs from the primary. I'm backing up a file system to disk and the files are small and there are lots of directories. The overall problem seems to be the total number of files. When I had ~375,000 files, the total rsync time was under a minute. With ~425,000 files, the total rsync time is 10 minutes. Last Friday when we were at 425,000 files, the rsync time was 10 minutes. Today I was able to delete 50,000 unneeded files and the rsync time went back down to under a minute. So why the huge change in total rsync time for a somewhat small change in total number of files? I'm afraid that as the total number of files keeps increasing that the total rsync time is going to go exponential. I turn the --progress flag on, and the time is rougly divided up evenly between building the file list and looking thru the file list. The files themselves are really small (~16K) and I'm not seeing any problem with anything other than how long it takes rsync to make a pass thru all the files. I do use the --delete option. The servers are Dell 2950s, builtin RAID 10 disks and 4Gig of RAM. OS is Centos 5.1. I'm running rsync 2.6.8 protocol version 29. This smells to me like some sort of caching problem. Is there something in the kernel or rsync itself that I can tweek? I'm no expert, but I suggest using rsync 3.x (3.0.6 for example), it doesn't keep the as much information of the filelist in memory. It's probably swapping to disk, because of the large list and that significantly slows down the performance of the whole machine(s). Have a look at the output of the 'vmstat 2' command on both machines while it's busy, specifically look at the caption that says 'swap', it has a 'si' and 'so' column below it. 'si' means reading from swap/disk and 'so' means writing to swap/disk. You can try it out fairly easily, especially if you don't use rsync for anything else. If you can't find a package, just building it is possible an option: cd /usr/src wget http://rsync.samba.org/ftp/rsync/rsync-3.0.6.tar.gz tar -zxvf rsync-3.0.6.tar.gz nice ./configure nice make That should work (atleast if you have gcc and make and possible other things already installed). And instead of calling rsync, you call /usr/src/rsync-3.0.6/rsync if you just want te test it first without installing. You'll have to do it on both machines ofcourse. If you are not sure you want to make any changes, with an unsupported binary, you can use: -n that would make rsync not write changes to disk. Hope these instructions help. Thanks, Mike -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: difference between --no-implied-dir abd --keep-dirlinks ?
On Mon, Jun 29, 2009 at 10:33:34AM +0530, Jignesh Shah wrote: I have gone through the rsync documentation and also tested same both --no-implied-dir and --keep-dirlinks options. My question is what is the difference between these options in below scenario? Implied directories are those that are above the transfer area but included as a part of the path due to the use of -R (--relative). By default all implied directories get sent with their directory attributes, and the destination is forced to match that. If you want the implied paths to differ, --no-implied-dir will leave them alone. Using --keep-dirlinks also affects the transfer area, so that you can put a symlink to a directory anyplace that the sender has a dir. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [rsync] include-from with different source points
On Mon, Jun 29, 2009 at 09:43:02AM -0400, GravyFace wrote: sudo rsync --progress -avz --include-from=/home/gravyface/backup.list / /mnt/usbbackup/localbackup/ If you drop the '*'s from the lines in the file and switch to using --files-from, that should do what you were expecting. You still need to specify the / source because files-from files are all relative to the source. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: difference between --no-implied-dir abd --keep-dirlinks ?
Thanks Wayne; I understood your point but two more questions arise from details given by you is, if I have --keep-dirlinks specified but not --no-implied-dir then also it sends directory attributes, and the destination is forced to match that results into losing destination symlinks? I have tested this and noticed that it doesn't remove destination symlinks. The other question is, if I specified --no-implied-dir then would rsync synch symlinks and directory entries if same are not available at destination? I have tested this but still confusing in differentiating these two options. I am not able to find any differences. Could you help me out with one real example that show the difference between these options? Jignesh On Tue, Jun 30, 2009 at 9:09 PM, Wayne Davison way...@samba.org wrote: On Mon, Jun 29, 2009 at 10:33:34AM +0530, Jignesh Shah wrote: I have gone through the rsync documentation and also tested same both --no-implied-dir and --keep-dirlinks options. My question is what is the difference between these options in below scenario? Implied directories are those that are above the transfer area but included as a part of the path due to the use of -R (--relative). By default all implied directories get sent with their directory attributes, and the destination is forced to match that. If you want the implied paths to differ, --no-implied-dir will leave them alone. Using --keep-dirlinks also affects the transfer area, so that you can put a symlink to a directory anyplace that the sender has a dir. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: how to keep user/group information when rsyncing to windows for restoring back to
I am guessing that you would like to use rsync to create a bootable backup. It may also be worth while spending some time looking at CloneZilla or even dd? Dd is a really great tool. You know a copy of the device is bit for bit. You can then use rsync to make incremental copies of these directories which regularly change. In this situation you can restore the machine and then rsync any changes to the system, reboot and you should be up and running. If the system can be down for a while then there are more options, it should be possible to configure the machine to boot from the network each night perform the backup and then start up again. Then you would have a nightly backup which can be easily restored back onto the system. I have not setup any scripting with CloneZillia. However, I guess this is possible. I would be interested to know if anyone has a setup like this or just what people think. The advantage with this approach is that you will have a fully bootable image taken from the machine each night. The disadvantage is that the system may be down for a while depending upon how much data is on the system. I want to backup the who linux system to a windows system. The backup will not be used in windows and will only be used to restore to linux system later. So I want to know is it possible to use rsync to keep the user/ group and other permission information in windows file system so I can restore these information when restoring from windows to Linux? Perhaps --fake-super will help. See the manpage for details. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to get first level folder names with rsync commands?
Dear List, I set up a NetBackup destination on my server. And I backup test1 to NetBackup/test1, test2 to NetBackup/test2. Now I wanna know how many folders that have been backuped. It should be 2 folders, with the name of test1 and test2. But I don't know if there is any way to do so? Any advice and help is well appreciated. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to first level folder names with rsync commands?
Dear List, I set up a NetBackup destination on my server. And I backup test1 to NetBackup/test1, test2 to NetBackup/test2. Now I wanna know how many folders that have been backuped. It should be 2 folders, with the name of test1 and test2. But I don't know if there is any way to do so? Any advice and help is well appreciated. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How to get first level folder names with rsync commands?
Something like ls -d NetBackup/* | wc should be informative. (assuming something unixy, of course -- cygwin stuff might work) -Original Message- From: rsync-bounces+tony=servacorp@lists.samba.org [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On Behalf Of Daniel.Li Sent: Wednesday, July 01, 2009 12:17 AM To: Rsync Subject: How to get first level folder names with rsync commands? Dear List, I set up a NetBackup destination on my server. And I backup test1 to NetBackup/test1, test2 to NetBackup/test2. Now I wanna know how many folders that have been backuped. It should be 2 folders, with the name of test1 and test2. But I don't know if there is any way to do so? Any advice and help is well appreciated. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How to get first level folder names with rsync commands?
Well, what I mean is running some rsync command, and get a list something like. $ rsync options destination List: test1 test2 Is it possible with rsync options? On Wed, 2009-07-01 at 00:23 -0500, Tony Abernethy wrote: Something like ls -d NetBackup/* | wc should be informative. (assuming something unixy, of course -- cygwin stuff might work) -Original Message- From: rsync-bounces+tony=servacorp@lists.samba.org [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On Behalf Of Daniel.Li Sent: Wednesday, July 01, 2009 12:17 AM To: Rsync Subject: How to get first level folder names with rsync commands? Dear List, I set up a NetBackup destination on my server. And I backup test1 to NetBackup/test1, test2 to NetBackup/test2. Now I wanna know how many folders that have been backuped. It should be 2 folders, with the name of test1 and test2. But I don't know if there is any way to do so? Any advice and help is well appreciated. Thanks. -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
