Odd behavior
Hi Y'all, I'm seeing some interesting behavior that I was hoping someone could shed some light on. Basically I'm trying to rsync a lot of files, in a series of about 60 rsyncs, from one server to another. There are about 160 million files. I'm running 3 rsyncs concurrently to increase the speed, and as each one finishes, another starts, until all 60 are done. The machine I'm initiating the rsyncs on has 48GB RAM. This is CentOS linux 5.4, kernel revision 2.6.18-164.15.1.el5. Rsync version 3.0.5 (on both sides). I was able to rsync all the data over to the new machine. But, because there was so much data, I need to run the rsyncs again to catch data that changed during the last rsync run. It sort of hangs midway through. What happens is that as the rsyncs run, the memory usage on the machine slowly creeps up, using quite a bit of RAM, which is odd because I thought the rsyncs were counting files incrementally, to reduce RAM impact. But, looking at top, the rsync processes aren't using much RAM at all: top - 12:22:10 up 1 day, 27 min, 1 user, load average: 46.85, 46.37, 44.97 Tasks: 309 total, 8 running, 301 sleeping, 0 stopped, 0 zombie Cpu(s): 1.0%us, 13.8%sy, 0.0%ni, 84.9%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 49435196k total, 34842524k used, 14592672k free, 141748k buffers Swap: 10241428k total,0k used, 10241428k free,49428k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 7351 root 25 0 19892 9.8m 844 R 100.1 0.0 552:58.55 rsync 9084 root 16 0 13108 2904 820 R 100.1 0.0 299:24.59 rsync 4759 root 0 -20 1447m 94m 15m S 29.9 0.2 667:34.21 mmfsd 9539 root 16 0 30136 19m 820 R 6.3 0.0 6:29.28 rsync 9540 root 15 0 271m 46m 260 S 0.3 0.1 0:12.13 rsync 10047 root 15 0 10992 1212 768 R 0.3 0.0 0:00.01 top 1 root 15 0 10348 700 592 S 0.0 0.0 0:02.15 init ...etc... But nevertheless, 34GB RAM is in use. But what really kills things is that at some point, each rsync all of a sudden ramps up to 100% CPU usage, and the all activity for that rsync essentially stops. In the above example, 2 of the 3 rsyncs are in that 100% CPU state, while the third rsync is only at 6.3%, but that is the one actually doing something. In some cases all 3 rsyncs get to 100%, and they all stall, there is no network traffic on the NIC at all and they don't progress. Now mostly what they are doing is counting files, since most of the files are the same on both sides, but there are just so many files (160 million). I don't seem to be out of memory, but I don't know why rsync would go to 100% CPU and just stall. I am rsyncing from an rsync server to my local server, with commands similar to this: rsync -a --delete rsync://encodek-0-4/data/genomes/ /hive/data/genomes/ Again, both sides at version 3.0.5. Nothing fancy or special. I have confirmed that it does count the files incrementally by running a few manually, it does report getting incremental file list Any ideas why the processes go to 100% CPU and then stall? I should also note that the initial run of rsyncs, where it was actually copying a ton of data, did not seem to have this problem, but now that the data is there and I'm rsyncing again, it seems to have this problem. Is it somehow related to the fact that it is mostly comparing a ton of files very quickly but not actually copying many of them? Thanks for any ideas! -erich -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Odd behavior
Well, I solved this problem myself, it seems. It was not an rsync problem, per se, but it's interesting anyway on big filesystems like this so I'll outline what went down: Because my rsyncs were mostly just statting millions of files very quickly, RAM filled up with inode cache. At a certain point, the kernel stopped allowing new cache entries to be added to the slab memory it had been using, and was slow to reclaim memory on old, clean inode cache entries. So it basically slowed the I/O of the computer to barely anything. Slab memory can be checked by looking at the /proc/meminfo file. If you see that slab memory is using up a fair portion of your total memory, run the 'slabtop' program to see the top offenders. In my case, it was the filesystem that was screwing me (by way of the kernel). I was able to speed up the reclaiming of clean, unused inode cache entries by tweaking this in the kernel: # sysctl -w vm.vfs_cache_pressure=1 The default value for that is 100, where higher values release memory faster for dentries and inodes. It helped, but my rsyncs were still faster than it was, and it didn't help that much. What really fixed it was this: # echo 3 /proc/sys/vm/drop_caches That effectively clears ALL dentry and inode entries from slab memory immediately. When I did that, memory usage dropped from 35GB to 500MB, my rsyncs fired themselves up again magically, and the computer was responsive again. Slab memory began to fill up again of course, as the rsyncs were still going. But slowly. For this edge case, I'm just going to configure a cron job to flush the dentry/inode cache every five minutes or so. But things look much better now! A word of warning for folks rsyncing HUGE numbers of files under linux. ;) As a side note, Solaris does not seem to have this problem, presumably because the kernel handles inode/dentry caching in a different way. -erich Erich Weiler wrote: Hi Y'all, I'm seeing some interesting behavior that I was hoping someone could shed some light on. Basically I'm trying to rsync a lot of files, in a series of about 60 rsyncs, from one server to another. There are about 160 million files. I'm running 3 rsyncs concurrently to increase the speed, and as each one finishes, another starts, until all 60 are done. The machine I'm initiating the rsyncs on has 48GB RAM. This is CentOS linux 5.4, kernel revision 2.6.18-164.15.1.el5. Rsync version 3.0.5 (on both sides). I was able to rsync all the data over to the new machine. But, because there was so much data, I need to run the rsyncs again to catch data that changed during the last rsync run. It sort of hangs midway through. What happens is that as the rsyncs run, the memory usage on the machine slowly creeps up, using quite a bit of RAM, which is odd because I thought the rsyncs were counting files incrementally, to reduce RAM impact. But, looking at top, the rsync processes aren't using much RAM at all: top - 12:22:10 up 1 day, 27 min, 1 user, load average: 46.85, 46.37, 44.97 Tasks: 309 total, 8 running, 301 sleeping, 0 stopped, 0 zombie Cpu(s): 1.0%us, 13.8%sy, 0.0%ni, 84.9%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 49435196k total, 34842524k used, 14592672k free, 141748k buffers Swap: 10241428k total,0k used, 10241428k free,49428k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 7351 root 25 0 19892 9.8m 844 R 100.1 0.0 552:58.55 rsync 9084 root 16 0 13108 2904 820 R 100.1 0.0 299:24.59 rsync 4759 root 0 -20 1447m 94m 15m S 29.9 0.2 667:34.21 mmfsd 9539 root 16 0 30136 19m 820 R 6.3 0.0 6:29.28 rsync 9540 root 15 0 271m 46m 260 S 0.3 0.1 0:12.13 rsync 10047 root 15 0 10992 1212 768 R 0.3 0.0 0:00.01 top 1 root 15 0 10348 700 592 S 0.0 0.0 0:02.15 init ...etc... But nevertheless, 34GB RAM is in use. But what really kills things is that at some point, each rsync all of a sudden ramps up to 100% CPU usage, and the all activity for that rsync essentially stops. In the above example, 2 of the 3 rsyncs are in that 100% CPU state, while the third rsync is only at 6.3%, but that is the one actually doing something. In some cases all 3 rsyncs get to 100%, and they all stall, there is no network traffic on the NIC at all and they don't progress. Now mostly what they are doing is counting files, since most of the files are the same on both sides, but there are just so many files (160 million). I don't seem to be out of memory, but I don't know why rsync would go to 100% CPU and just stall. I am rsyncing from an rsync server to my local server, with commands similar to this: rsync -a --delete rsync://encodek-0-4/data/genomes/ /hive/data/genomes/ Again, both sides at version 3.0.5. Nothing fancy or special. I have confirmed that it does count the files incrementally by running a few manually, it
Re: Odd behavior
Hi, On Thu, 22 Apr 2010, Erich Weiler wrote: Well, I solved this problem myself, it seems. It was not an rsync problem, per se, but it's interesting anyway on big filesystems like this so I'll outline what went down: Because my rsyncs were mostly just statting millions of files very quickly, RAM filled up with inode cache. At a certain point, the kernel stopped allowing new cache entries to be added to the slab memory it had been using, and was slow to reclaim memory on old, clean inode cache entries. So it basically slowed the I/O of the computer to barely anything. Slab memory can be checked by looking at the /proc/meminfo file. If you see that slab memory is using up a fair portion of your total memory, run the 'slabtop' program to see the top offenders. In my case, it was the filesystem that was screwing me (by way of the kernel). I was able to speed up the reclaiming of clean, unused inode cache entries by tweaking this in the kernel: # sysctl -w vm.vfs_cache_pressure=1 The default value for that is 100, where higher values release memory faster for dentries and inodes. It helped, but my rsyncs were still faster than it was, and it didn't help that much. What really fixed it was this: # echo 3 /proc/sys/vm/drop_caches That effectively clears ALL dentry and inode entries from slab memory immediately. When I did that, memory usage dropped from 35GB to 500MB, my rsyncs fired themselves up again magically, and the computer was responsive again. Slab memory began to fill up again of course, as the rsyncs were still going. But slowly. For this edge case, I'm just going to configure a cron job to flush the dentry/inode cache every five minutes or so. But things look much better now! A word of warning for folks rsyncing HUGE numbers of files under linux. ;) As a side note, Solaris does not seem to have this problem, presumably because the kernel handles inode/dentry caching in a different way. Thanks for your report. Really a mail to keep for future. Viele Gruesse Eberhard Moenkeberg (emoe...@gwdg.de, e...@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoe...@gwdg.de Tel.: +49 (0)551 201-1551 - Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL:http://www.gwdg.de E-Mail: g...@gwdg.de Tel.: +49 (0)551 201-1510Fax:+49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Bernhard Neumair Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 - -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Odd behavior in an exclude-file
On Monday 22 December 2008 03:38:20 pm Matt McCutchen wrote: On Fri, 2008-12-19 at 10:01 -0600, Roy F. Cabaniss wrote: rsync -avl --stats --progress --timeout=300 --exclude-from /home/foo/bin/exclude.txt /home /mnt/sdc2 Since there are, as with any backups, files I don't want to bother backing up I created an exclude file and stored it in my bin. - /home/foo/vmware - /home/foo/.kde/share/apps/kmail/mail/spam/* - /home/foo/.kde/share/apps/kmail/mail/.spam.directory/* - /home/foo/.mozilla/firefox/tigy4u04.default/Cache - /home/foo/packages - /home/foo/.cxgames - /home/foo/.cxoffice - /home/foo/.beagle - /home/foo/downloads/images With the rsync script in the /home/foo/bin I ran the script with myself as root in the root directory. Which leads to the reason for this letter. The exclude file appears to have worked, with one exception. Much to my surprise, the images directory was in the backup along with its contents. I can assure you that the path is as given. Is there any idea why I got that one directory in the backup? Perhaps run od -t x1c on the exclude file to see if there is anything funny about the last line. E.g., if there is a trailing space, that would prevent the rule from matching. If that isn't it, you could see if the last rule works if you move it earlier in the exclude file, or if you change it to just - images (which probably excludes more than you want but could shed light on the problem). Just to let everyone know: I made sure there were a couple of spare line feeds after the last line. I added a dummy exclude directory after the images line. It worked fine then. I don't know which of the two things I did was what made it work ok. My guess would be the dummy exclude line because I did already have a linefeed after the last exclude line. Thanks for the help. -- Roy F. Cabaniss http://www.housedraco.org At least one attached file is my gpg signature. If you don't know how to open/use the file, don't worry about it. It is included for those who DO know and understand such things. signature.asc Description: This is a digitally signed message part. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Odd behavior in an exclude-file
Try putting some additional line breaks at the end of your file. I am not sure if this will solve your issues. Give it a go and report back if this resolves the issue. Hope this helps I decided the most secure way to deal with backup/firewall issues between my work and home was to encrypt a portable hard drive and make it my backup. Lug it back and forth and sync as appropriate. So I wrote myself a little rsync script which grabs all the files I think of as taking work to recreate. Which is given as follows: rsync -avl --stats --progress --timeout=300 --exclude-from /home/ foo/bin/exclude.txt /home /mnt/sdc2 rsync -avl --stats --progress --timeout=300 --exclude-from /home/ foo/bin/exclude.txt /etc /mnt/sdc2/beast rsync -avl --stats --progress --timeout=300 --exclude-from /home/ foo/bin/exclude.txt /srv /mnt/sdc2/beast Since there are, as with any backups, files I don't want to bother backing up I created an exclude file and stored it in my bin. - /home/foo/vmware - /home/foo/.kde/share/apps/kmail/mail/spam/* - /home/foo/.kde/share/apps/kmail/mail/.spam.directory/* - /home/foo/.mozilla/firefox/tigy4u04.default/Cache - /home/foo/packages - /home/foo/.cxgames - /home/foo/.cxoffice - /home/foo/.beagle - /home/foo/downloads/images With the rsync script in the /home/foo/bin I ran the script with myself as root in the root directory. Which leads to the reason for this letter. The exclude file appears to have worked, with one exception. I don't have the vmware directory, the link to packages was not followed, the crossover directories are not in the backup ect. All very good and as hoped. The exception is in the very last line. The very last line of the exclude file is the directory that contains iso images I downloaded for whatever reason. Things like my latest image for my distro and such. I did NOT want to back those up since in the event of catastrophe I would simply redownload them and the files are rather sizeable. Much to my surprise, the images directory was in the backup along with its contents. I can assure you that the path is as given. Is there any idea why I got that one directory in the backup? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Odd behavior in an exclude-file
On Fri, 2008-12-19 at 10:01 -0600, Roy F. Cabaniss wrote: rsync -avl --stats --progress --timeout=300 --exclude-from /home/foo/bin/exclude.txt /home /mnt/sdc2 Since there are, as with any backups, files I don't want to bother backing up I created an exclude file and stored it in my bin. - /home/foo/vmware - /home/foo/.kde/share/apps/kmail/mail/spam/* - /home/foo/.kde/share/apps/kmail/mail/.spam.directory/* - /home/foo/.mozilla/firefox/tigy4u04.default/Cache - /home/foo/packages - /home/foo/.cxgames - /home/foo/.cxoffice - /home/foo/.beagle - /home/foo/downloads/images With the rsync script in the /home/foo/bin I ran the script with myself as root in the root directory. Which leads to the reason for this letter. The exclude file appears to have worked, with one exception. Much to my surprise, the images directory was in the backup along with its contents. I can assure you that the path is as given. Is there any idea why I got that one directory in the backup? Perhaps run od -t x1c on the exclude file to see if there is anything funny about the last line. E.g., if there is a trailing space, that would prevent the rule from matching. If that isn't it, you could see if the last rule works if you move it earlier in the exclude file, or if you change it to just - images (which probably excludes more than you want but could shed light on the problem). -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Odd behavior in an exclude-file
I decided the most secure way to deal with backup/firewall issues between my work and home was to encrypt a portable hard drive and make it my backup. Lug it back and forth and sync as appropriate. So I wrote myself a little rsync script which grabs all the files I think of as taking work to recreate. Which is given as follows: rsync -avl --stats --progress --timeout=300 --exclude-from /home/foo/bin/exclude.txt /home /mnt/sdc2 rsync -avl --stats --progress --timeout=300 --exclude-from /home/foo/bin/exclude.txt /etc /mnt/sdc2/beast rsync -avl --stats --progress --timeout=300 --exclude-from /home/foo/bin/exclude.txt /srv /mnt/sdc2/beast Since there are, as with any backups, files I don't want to bother backing up I created an exclude file and stored it in my bin. - /home/foo/vmware - /home/foo/.kde/share/apps/kmail/mail/spam/* - /home/foo/.kde/share/apps/kmail/mail/.spam.directory/* - /home/foo/.mozilla/firefox/tigy4u04.default/Cache - /home/foo/packages - /home/foo/.cxgames - /home/foo/.cxoffice - /home/foo/.beagle - /home/foo/downloads/images With the rsync script in the /home/foo/bin I ran the script with myself as root in the root directory. Which leads to the reason for this letter. The exclude file appears to have worked, with one exception. I don't have the vmware directory, the link to packages was not followed, the crossover directories are not in the backup ect. All very good and as hoped. The exception is in the very last line. The very last line of the exclude file is the directory that contains iso images I downloaded for whatever reason. Things like my latest image for my distro and such. I did NOT want to back those up since in the event of catastrophe I would simply redownload them and the files are rather sizeable. Much to my surprise, the images directory was in the backup along with its contents. I can assure you that the path is as given. Is there any idea why I got that one directory in the backup? -- Roy F. Cabaniss http://www.housedraco.org At least one attached file is my gpg signature. If you don't know how to open/use the file, don't worry about it. It is included for those who DO know and understand such things. signature.asc Description: This is a digitally signed message part. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
odd behavior on remote
I've been using rsync for some time (years) to generate many hardlink snapshots per day; but I'm seeing an odd new problem today. the remote/destination host gets a file list from the source machine via ssh, and begins to write files until it hangs. On this run only one file was transferred; on other runs many screenfulls went across + rsync --recursive --links --perms --times --group --owner --devices --specials --numeric-ids --protocol=29 --verbose --progress --exclude tmp --exclude *.tmp --exclude spool --exclude *.core --exclude *.boot --exclude *.filepart --exclude *.lock --exclude *.nobak --exclude .RDATA --exclude /repo --exclude /sandbox --exclude /soft/dist --link-dest=/ub0/bk/1/2008.04.29.0010.00//center //center/ lou.grid://ub0/bk/0/2008.05.08.1051.49//center/ building file list ... 580347 files to consider comp/GridTest/101v6.csv the dest host is rsync version 2.6.9 protocol version 29 and the source host is rsync version 2.6.9 protocol version 29 in the above run I specified the protocol because I assumed the src host had a newer version. the src host has just been setup, but the destination host has been receiving these snapshots for a while. After the hang, the destination host seems in perfect order (no login or disk or observable problems). I see the ssh connection from source to destination is still open but the remote rsync pids have all ended with no indication of error -- there is no rsync in the process tree at all; below, the rsync shell is/was 26770, now with no children... | | |-+= 03778 root sshd: [EMAIL PROTECTED] | | | \--= 17062 root -ksh | | \--= 26770 root sshd: [EMAIL PROTECTED] | |-+= 00594 root nfsd: master | | |--- 00475 root nfsd: server | | |--- 00601 root nfsd: server The only problem I see with the source host is the rsync command doesn't complete. What can I check? What is going on here??? // George -- George Georgalis, information system scientist IXOYE -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: odd behavior on remote
On Thu 08 May 2008 at 11:52:08 AM -0400, George Georgalis wrote: I've been using rsync for some time (years) to generate many hardlink snapshots per day; but I'm seeing an odd new problem today. OOOh, nevermind... FilesystemSize Used Avail Capacity iusedifree %iused Mounted on /dev/raid0h 312G 312G -16G 105% 7502943 3339497518% /ub0 I'm out of disk space ;) // George -- George Georgalis, information system scientist IXOYE -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Odd behavior with --detect-renamed
On Fri, 2007-12-28 at 17:31 +0100, Erik Pettersson wrote: I'm trying out the 'detect-renamed'-patch, and I've encountered some odd behavior. Basicly, what I've noticed is that if I move a file into a newly created directory (which is what happens if I rename a directory, for example), the file isn't detected as renamed. However, if I manually create the new directory on the target-side, the file is correctly detected as a rename, and isn't transfered over the network. Here I've moved 'file' into a newly created directory 'dir2', which isn't available on the target-side. $ find src src src/dir1 src/dir1/dir2 src/dir1/dir2/file $ find dst dst dst/dir1 dst/dir1/file $ rsync -avv --detect-renamed -e ssh src/dir1 localhost:dst [...] total: matches=0 hash_hits=0 false_alarms=0 data=4544512 The file isn't detected as a rename. I see what the problem is. When the receiving rsync considers dst/dir1/file for a rename, dst/dir/dir2 does not yet exist, so rsync cannot create the partial dir for dst/dir/dir2 in order to stage the rename. One workaround would be to do a preliminary run with --include='*/' --exclude='*' to create all the necessary directories in the destination before doing the main run. One possible fix would be to call create_directory_path to ensure that the destination directory that is the target of the rename exists before handling the partial dir. Unfortunately, create_directory_path will give any created directories default permissions instead of flist ones, which could be bad. To avoid this problem, rsync could look up each intermediate directory in the flist and, if it is found, give it 700 permissions for later fixing (like the incremental recursion code does). This is awkward, which I believe is a sign that the reverse design of pre-scanning the destination files and looking up each source file in turn may be better. This design was proposed by Charles Perreault to me off-list and quoted in: http://lists.samba.org/archive/rsync/2007-November/019067.html Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Odd behavior with --detect-renamed
Hello, I'm totally new to this list, so I hope I don't break all the rules. :) I've looked through the archives (and google), and I really can't find the answer to my question. I'm trying out the 'detect-renamed'-patch, and I've encountered some odd behavior. I've applied the patch to both rsync-2.6.9 and rsync-3.0.0pre7, and it's the same behavior. Basicly, what I've noticed is that if I move a file into a newly created directory (which is what happens if I rename a directory, for example), the file isn't detected as renamed. However, if I manually create the new directory on the target-side, the file is correctly detected as a rename, and isn't transfered over the network. Here I've moved 'file' into a newly created directory 'dir2', which isn't available on the target-side. $ find src src src/dir1 src/dir1/dir2 src/dir1/dir2/file $ find dst dst dst/dir1 dst/dir1/file $ rsync -avv --detect-renamed -e ssh src/dir1 localhost:dst [...] total: matches=0 hash_hits=0 false_alarms=0 data=4544512 The file isn't detected as a rename. If I create 'dir2' manually on the target-side, it is detected correctly: $ find src src src/dir1 src/dir1/dir2 src/dir1/dir2/file $ find dst dst dst/dir1 dst/dir1/file dst/dir1/dir2 $ rsync -avv --detect-renamed -e ssh src/dir1 localhost:dst [...] found renamed: dir1/file = dir1/dir2/file [...] total: matches=2136 hash_hits=2136 false_alarms=0 data=0 Is this a bug (I think so), or should it be like this? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Odd behavior with rsync/ssh/--delete
I've just about googled my brains out over this one, and banged heads with several other SA buddies. I have a nightly rsync of a DMZ system (Solaris 8 SPARC[1]) to an internal system (RedHat ES 3.0 [2]). The internal system runs a cron job and pulls changes off of the DMZ system via ssh. (To be honest, I've also seen this going between two Solaris systems.) However, my syncs are much bigger then they should be. I'm getting a bunch more than I expect - files that haven't changed in a long time are being deleted and re-sync'd. Here's an example of the command being used: rsync -alvx --delete --rsh=ssh [EMAIL PROTECTED]:/ /backups0/hosts//rsync/ (I use -x to avoid crossing filesystems, as I only back up certain ones. The refers to the hostname of the DMZ system.) What I see are files like this getting hit every night in the log: (real user name blanked.) [...] deleting home/xuserx/public_html/vat5.JPG [...] Then later the file is written. If you look at the file on the host: # ls -la vat5.JPG -rw-rw-r-- 1 xuserx xgroupx 39816 Jan 3 2000 vat5.JPG This is strange. And it's happening for thousands of files. The systems have the same date and time, I checked that out... I'm at wit's end on this one. If anybody has any ideas, I'd be happy to hear them. Also, I can provide more output if desired. Cheers, -Pete Wargo [1] rsync version 2.5.6 protocol version 26. Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no IPv6, 64-bit system inums, 64-bit internal inums [2] rsync version 2.5.6 protocol version 26. Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, IPv6, 64-bit system inums, 64-bit internal inums -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
