RE: --delete doesn't work when transferring many files
Is it possible that the destination directory is filling up? I believe rsync saves deletes until the end, so that it knows there are no IO error conditions that would cause it to exit without deleting. Filling up the destination filesystem is probably one of those error conditions. Another possibility would be permissions problems on the destination preventing rsync from writing to the target directory. Try adding another v to your command line (-rtvv) to get more verbosity; rsync may tell you why it is skipping the deletes. You could add --ignore-errors to force the --delete, but I would recommend finding the cause of the problem first. -Original Message- From: William R. Dickson [mailto:[EMAIL PROTECTED]] Sent: Tuesday, July 23, 2002 9:30 PM To: [EMAIL PROTECTED] Subject: --delete doesn't work when transferring many files This is a curious thing. I'm syncing two directories, containing some 250 subdirectories totalling around 11,000 files. This: rsync --password-file=/usr/local/etc/rsync/.passwd --delete -rtv --exclude-from=/usr/local/etc/rsync/excludes /usr/local/sourcedir [EMAIL PROTECTED]::modulename ...works fine if both directories are already nearly in sync. Deleting a few files from the source causes matching deletions on the destination. But recently I made major changes to the source directory, which should have resulted in nearly everything being deleted from the destination server. However, it doesn't -- it simply starts uploading files. If I change -rtv to -rtvn, rsync correctly reports a whole slew of planned deletions -- but as soon as I try to run it for real, it simply fails to do the actual deleting. If it helps at all, the source system is a NetBSD/MacPPC machine, and the destination is a FreeBSD/i386 machine. Any ideas? I searched the list archives and google, but was unable to find any similar issues. Thanks, -Bill -- William R. Dickson -- Consuming the Earth's limited resources since 1968 [EMAIL PROTECTED]http://www.seavoy-dickson.org/ But I'm the luckiest guy on the lower east side 'Cuz I've got wheels, and you wanna go for a ride - The Magnetic Fields -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Rsync --delete does not work
Try adding -v to your rsync command to get more verbose output, perhaps giving you an idea why --delete is not working as expected. --delete works for me, but there are situations where rsync will ignore --delete for safety (such as if certain errors occur). Also, remember that * is expanded by your shell, before rsync sees the command line. So if your /data/transfer contained files a, b, c, and d, and you deleted c, rsync would see your command line as: rsync -a --delete a b d /data/exp_dir In other words, you've told rsync not to sync c, so it would have no reason to delete it at the destination directory. Try using . instead of * to get more reliable results. Doug -Original Message- From: g dm [mailto:[EMAIL PROTECTED]] Sent: Tuesday, July 23, 2002 11:55 AM To: [EMAIL PROTECTED] Subject: Rsync --delete does not work Hi I just started to use rsync samba to backup some windows NT 98 shared directories to a linux machine linux: debian 2.2.4 rsync: 2.2.5 samba: 2.2.3a-6 I first mount the windows directory on the linux and then do a local rsync. ---command on linux umount /data/transfer mount -t smbfs -o username=user,password=passwd //ntserver/exp_dir /data/transfer cd /data/transfer rsync -a --delete * /data/exp_dir umount /data/transfer --- this works fine, except for deleted files. Files that got deleted on the windows remain on the linux side, even after repeated rsync's This is what the --delete option is supposed to do. I've tested this several times. the samba mount show the correct state, i.e. without the deleted files. Rsync does not notice they're gone. I've seen/read a lot of posts in the google archives concerning this --delete option. Most of them came from this mailing list. They all seem to suggest it really should work. So, what did I do wrong? best regards glenn -- __ Sign-up for your own FREE Personalized E-mail at Mail.com http://www.mail.com/?sr=signup Save up to $160 by signing up for NetZero Platinum Internet service. http://www.netzero.net/?refcd=N2P0602NEP8 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Largest file system being synced
I am currently syncing 1.3 terabytes of files using rsync. This is spread across about 12 filesystems all on the same server. Unfortunately we are planning to move away from rsync because it is taking too long to run and it takes up too much memory (some of the rsync processes take up 1.5 GB of RAM -- if we get two of those running at once, the server dies). We have been very happy with rsync but it has recently reached a critical mass where it can no longer handle the number of files we are trying to sync in a timely manner. But if you are looking for the largest rsync site, we might be a contender. :) FYI, we also use (and plan to continue to use) rsync for several smaller mirroring operations. I'm interested in large file system replication capability of rsync. Could some of you people who use it share how large their mirroring is? What would you say is the largest sized site being mirrored using rsync? Thanks! JP -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Largest file system being synced
We're planning to move to Veritas Volume Replicator. It has the advantage of working at the filesystem level, so whenever a write is done on the primary site, the same write is automatically done on the mirror site. For what I am doing here, it should (hopefully!) work a lot better than rsync because it runs continuously, and it doesn't have the startup overhead of building a file list, which is the biggest problem with rsync when you are dealing with large groups of files. -Original Message- From: Breedlove, Robert [mailto:[EMAIL PROTECTED]] Sent: Thursday, June 27, 2002 11:46 AM To: Granzow, Doug (NCI); '[EMAIL PROTECTED]' Subject: RE: Largest file system being synced What are you moving to? -Original Message- From: Granzow, Doug (NCI) [mailto:[EMAIL PROTECTED]] Sent: Thursday, June 27, 2002 8:31 AM To: '[EMAIL PROTECTED]' Subject: RE: Largest file system being synced I am currently syncing 1.3 terabytes of files using rsync. This is spread across about 12 filesystems all on the same server. Unfortunately we are planning to move away from rsync because it is taking too long to run and it takes up too much memory (some of the rsync processes take up 1.5 GB of RAM -- if we get two of those running at once, the server dies). We have been very happy with rsync but it has recently reached a critical mass where it can no longer handle the number of files we are trying to sync in a timely manner. But if you are looking for the largest rsync site, we might be a contender. :) FYI, we also use (and plan to continue to use) rsync for several smaller mirroring operations. I'm interested in large file system replication capability of rsync. Could some of you people who use it share how large their mirroring is? What would you say is the largest sized site being mirrored using rsync? Thanks! JP -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: memory requirements was RE: out of memory in build_hash_table
Well, for reference, here's what I'm running: The mirror is running Solaris 8, the source is running Solaris 7. Both are running the same version of rsync: rsync version 2.5.4 protocol version 26 Copyright (C) 1996-2002 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no IPv6, 64-bit system inums, 64-bit internal inums The command being run on the mirror is: /usr/local/bin/rsync -qau --delete --rsh=/usr/bin/ssh source:/share/abc/. /share/abc/ From what I've observed by running top while rsync is running, its memory usage appears to grow gradually, not exponentially. A rsync may take several hours to complete. (I have one running now that started over four hours ago. The filesystem contains 236 GB of data in 2.4 million files. It is currently taking up 1351MB of memory on the mirror server and 646M on the source server.) All filesystems are veritas filesystem, in case that is relevant. I saw someone on this list recently mentioned changing the block size. The rsync man page indicates this defaults to 700 (bytes?). Would a larger block size reduce memory usage, since there will be fewer (but larger) blocks of data, and therefore fewer checksums to store? You suggested setting ARENA_SIZE to 0... I guess this would be done like this? % ARENA_SIZE=0 ./configure Doug David Bolen [mailto:[EMAIL PROTECTED]] writes: Granzow, Doug (NCI) [[EMAIL PROTECTED]] writes: Hmm... I have a filesystem that contains 3,098,119 files. That's 3,098,119 * 56 bytes or 173,494,664 bytes (about 165 MB). Allowing for the exponential resizing we end up with space for 4,096,000 files * 56 bytes = 218 MB. But 'top' tells me the rsync running on this filesystem is taking up 646 MB, about 3 times what it should. Are there other factors that affect how much memory rsync takes up? I only ask because I would certainly prefer it used 218 MB instead of 646. :) Hmm, yes - I only mentioned the per-file meta-data overhead since that's the only memory user in the original note case, which was failing before it actually got the file list transferred, and it hadn't yet started computing any checksums. But there are definitely some other dynamic memory chunks. However, in general the per-file meta-data ought to be the major contributor to memory usage. I've attached an old e-mail of mine when I did some examining of memory usage for an older version of rsync (2.4.3) which I think it still fairly valid. I don't think it'll explain your significantly larger usage than expected. (A followup note corrected the first paragraph as rsync doesn't create any tree structures) Two possibilities I can think of have to do with the fact that the per-file overhead is handled by 'realloc'ing the space as it grows. It's possible that the sequence of events is such that some other allocation is being done in the midst of that growth which forces the next realloc to actually move the memory to gain more space, thus leaving a hole of unused memory that just takes up process space. Or, it's also possible that the underlying allocation library (e.g., the system malloc()) is itself performing some exponential rounding up in order to help prevent just such movement. I know that AIX used to do that, and even provided an environment variable way to revert to older behavior. What you might try doing is observing the process growth during the directory scanning phase and see how much memory actually gets used to that point in time - gauged either by observing client/server traffic for when the file list starts getting transmitted, or by enabling/adding some debugging output to rsync. I just peeked at the latest sources in CVS, and it looks like around version 2.4.6 the file list processing added some of it's own micro-management of memory for small strings, so there's something else going on there too, in theory to help avoid platform growth like mentioned in my last paragraph. So if you're using 2.4.6, you might try a later version to see if it improves things. Or if you're using a later version you might try rebuilding with ARENA_SIZE set to 0 to disable this code to see if your native platform handles it better somehow. -- David /- --\ \ David Bolen\ E-mail: [EMAIL PROTECTED] / | FitLinxx, Inc.\ Phone: (203) 708-5192| / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \- --/ - - - - - - - - - - - - - - - - - - - - - - - - - From: David Bolen [EMAIL PROTECTED] To: 'Lenny Foner' [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: RE: The out of memory problem with large numbers of files Date: Thu, 25 Jan 2001 13:25:43 -0500 Lenny Foner
memory requirements was RE: out of memory in build_hash_table
The key growth factor for the file construction is going to be per-file information that's about 56 bytes (last I looked) per file. However, the kicker is that the file storage block gets resized exponentially as you have more files. So for 385,000 files it'll actually be a 512,000 file block of about 30MB. (So yeah, I suppose an ~50 byte file chunk in memory growing as a power of 2 might average out close to 100 bytes/file as an estimate :-)) Hmm... I have a filesystem that contains 3,098,119 files. That's 3,098,119 * 56 bytes or 173,494,664 bytes (about 165 MB). Allowing for the exponential resizing we end up with space for 4,096,000 files * 56 bytes = 218 MB. But 'top' tells me the rsync running on this filesystem is taking up 646 MB, about 3 times what it should. Are there other factors that affect how much memory rsync takes up? I only ask because I would certainly prefer it used 218 MB instead of 646. :) Doug -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: large file error is now SIGUSR1 or SIGINT error
Well, I found my problem. After doing many trusses, modifying rsync source code to generate various debugging information, and learning my way around a few parts of rsync a lot better than an end-user should, I determined that the problem has nothing to do with rsync whatsoever. :) The problem was that the destination filesystem (a vxfs filesystem) was not configured to support large files. You were right, Dave, when you wrote The real problem is your write failure. I enabled large file support on vxfs and rsync is now happily copying files of all sizes. To answer my own questions (from the bottom of my last message, below) (and for the benefit of someone searching through the mailing list archive): 1) Yes, rsync supports files over 2GB when compiled on Solaris 7 or 8. 2) The problem was the destination filesystem (a Veritas File System) was not configured to support files over 2GB. 3) I didn't find a way to do this, but I no longer have any need to. Thanks, Doug -Original Message- From: Dave Dykstra [mailto:[EMAIL PROTECTED]] Sent: Tuesday, February 12, 2002 11:54 AM To: Granzow, Doug (NCI) Cc: '[EMAIL PROTECTED]' Subject: Re: large file error is now SIGUSR1 or SIGINT error The SIGUSR1 or SIGINT is just a secondary message in this case that you can ignore. The receiver side of rsync splits into two processes, and that's just the message that the second one prints after the first one kills it off because it had a problem. The real problem is your write failure. I don't have any experience with using 2GB files on Solaris 7 or 8 so hopefully somebody else can help you with that or you can figure out the problem yourself. The Solaris tools I distribute inside my company are all compiled on 2.5.1 (because I need to support users on that OS version and up) so I'm stuck with the 32 bit limit. - Dave Dykstra On Tue, Feb 12, 2002 at 11:31:55AM -0500, Granzow, Doug (NCI) wrote: I just ran this again and got this error: leelab/NCBI_Data_old/GenBank/htg write failed on leelab/NCBI_Data_old/GenBank/htg : Error 0 rsync error: error in file IO (code 11) at receiver.c(243) Received signal 16. (no core) rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(229) The command I am running is: /usr/local/bin/rsync -auv --delete --rsh=/usr/bin/ssh lpgfs104:/share/group/* /share/group/ An update on this problem... I get the error below (and the error I reported previously) when running rsync 2.5.2 compiled from source. I saw different behavior when I used the rsync 2.5.2 binary compiled on Solaris 2.5.1 by Dave Dykstra. That binary complained of Value too large for defined data type whenever it encountered a large file (over 2GB), but did not exit. The impression I got was that the Solaris 2.5.1 binary did not support or even try to support files over 2 GB, where the binary compiled on Solaris 7 or 8 *thinks* it can support large files but fails, since it exits as soon as it encounters the large file. So the problem still remains: rsync is dying when it encounters a large file. One person suggested using --exclude, but this only matches against file names, not file sizes. (I can't do --exclude=size2GB for example.) Questions I still have: - Is rsync supposed to support files 2GB on Solaris 7 and Solaris 8? - If so, what is causing the errors I am seeing? Is there something I can do at compile time? - If not, is there a way for it to skip large files gracefully so that at least the rsync process completes? leelab/NCBI_Data_old/GenBank/htg write failed on leelab/NCBI_Data_old/GenBank/htg : Error 0 rsync error: error in file IO (code 11) at receiver.c(243) Received signal 16. (no core) rsync: connection unexpectedly closed (23123514 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(140)
RE: error in rsync protocol on large file
An update on this problem... I get the error below (and the error I reported previously) when running rsync 2.5.2 compiled from source. I saw different behavior when I used the rsync 2.5.2 binary compiled on Solaris 2.5.1 by Dave Dykstra. That binary complained of Value too large for defined data type whenever it encountered a large file (over 2GB), but did not exit. The impression I got was that the Solaris 2.5.1 binary did not support or even try to support files over 2 GB, where the binary compiled on Solaris 7 or 8 *thinks* it can support large files but fails, since it exits as soon as it encounters the large file. So the problem still remains: rsync is dying when it encounters a large file. One person suggested using --exclude, but this only matches against file names, not file sizes. (I can't do --exclude=size2GB for example.) Questions I still have: - Is rsync supposed to support files 2GB on Solaris 7 and Solaris 8? - If so, what is causing the errors I am seeing? Is there something I can do at compile time? - If not, is there a way for it to skip large files gracefully so that at least the rsync process completes? leelab/NCBI_Data_old/GenBank/htg write failed on leelab/NCBI_Data_old/GenBank/htg : Error 0 rsync error: error in file IO (code 11) at receiver.c(243) Received signal 16. (no core) rsync: connection unexpectedly closed (23123514 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(140) -Original Message- From: Granzow, Doug (NCI) Sent: Monday, February 11, 2002 11:08 AM To: '[EMAIL PROTECTED]' Subject: error in rsync protocol on large file I'm trying to rsync a filesystem from a Solaris 7 server to a Solaris 8 server. I keep getting the following message when rsync attempts to update a particular file: write failed on leelab/NCBI_Data/GenBank/htg : Error 0 rsync error: error in file IO (code 11) at receiver.c(243) Received signal 16. (no core) rsync: connection unexpectedly closed (22672537 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(140) The file it is getting the error on is very large: -rw-rw-r-- 1 leemastaff4633417879 Apr 30 2001 htg # rsync --version rsync version 2.5.2 protocol version 26 Copyright (C) 1996-2002 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no IPv6, 64-bit system inums, 64-bit internal inums Does rsync simply not handle files this large? Is there something in config.h I can change to support large files (I see #undef _LARGE_FILES but the comment above it indicates it is only for AIX)? If I can't get rsync to copy large files, is there a way to get it to skip them, so that the rest of the files do get synced? Thanks Doug
large file error is now SIGUSR1 or SIGINT error
I just ran this again and got this error: leelab/NCBI_Data_old/GenBank/htg write failed on leelab/NCBI_Data_old/GenBank/htg : Error 0 rsync error: error in file IO (code 11) at receiver.c(243) Received signal 16. (no core) rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(229) The command I am running is: /usr/local/bin/rsync -auv --delete --rsh=/usr/bin/ssh lpgfs104:/share/group/* /share/group/ An update on this problem... I get the error below (and the error I reported previously) when running rsync 2.5.2 compiled from source. I saw different behavior when I used the rsync 2.5.2 binary compiled on Solaris 2.5.1 by Dave Dykstra. That binary complained of Value too large for defined data type whenever it encountered a large file (over 2GB), but did not exit. The impression I got was that the Solaris 2.5.1 binary did not support or even try to support files over 2 GB, where the binary compiled on Solaris 7 or 8 *thinks* it can support large files but fails, since it exits as soon as it encounters the large file. So the problem still remains: rsync is dying when it encounters a large file. One person suggested using --exclude, but this only matches against file names, not file sizes. (I can't do --exclude=size2GB for example.) Questions I still have: - Is rsync supposed to support files 2GB on Solaris 7 and Solaris 8? - If so, what is causing the errors I am seeing? Is there something I can do at compile time? - If not, is there a way for it to skip large files gracefully so that at least the rsync process completes? leelab/NCBI_Data_old/GenBank/htg write failed on leelab/NCBI_Data_old/GenBank/htg : Error 0 rsync error: error in file IO (code 11) at receiver.c(243) Received signal 16. (no core) rsync: connection unexpectedly closed (23123514 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(140)
