In my humble opinion, this problem with rsync growing a huge memory footprint when large numbers of files are involved should be #1 on the list of things to fix. It seems that every fifth post is a complaint about this problem! Sorry if this sounds like ungrateful whining, but I think rsync would be greatly improved if the overall way it processes multiple files were overhauled to be much more resource friendly.
John. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, May 17, 2002 01:44 PM To: C.Zimmermann Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: Rsync dies Yeah. You'll have to find a way to break the job up into smaller pieces. It's a pain, but I have a similar situation - 3M+ files in 130+Gb. I can't get the whole thing in one chunk, no matter how fast a server with however much memory, even on Gb ethernet (for the server). In my case, the filesystem is on NAS, and the NAS has only 100bT simplex (half-duplex, to some). I have some code that can be used to analyze your system before the sync, and choose directories containing no more than a maximum number of items below them. Iterating through the list and using -R can let you get the whole thing run, though --delete and -H become less certain (not dangerous, but if you don't name anything containing a deleted directory because it didn't come up on your list, youll never tell the destination to delete it, and if you have two hard links to the same file, but hit them in two seperate runs, you now have two copies on disk). Let me know if you want it. I'm sure you can figure how to modify it for your environment. Tim Conway [EMAIL PROTECTED] 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips, n9hmg on AIM perl -e 'print pack(nnnnnnnnnnnn, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), ".\n" ' "There are some who call me.... Tim?" "C.Zimmermann" <[EMAIL PROTECTED]> Sent by: [EMAIL PROTECTED] 05/17/2002 02:08 AM To: <[EMAIL PROTECTED]> cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject: Rsync dies Classification: I�m trying to rsync a 210 GB Filesystem with approx 1.500.000 Files. Rsync always dies after about 29 GB without any error messages. I�m Using rsync version 2.5.5 protocol version 26. Has anyone an idea ? Thank�s Clemens -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
