Mark: You are S.O.L. There's been a lot of discussion on the subject, and so far, the only answer is faster machines with more memory. For my own application, I have had to write my own system, which can be best described as find, sort, diff, grep, cut, tar, gzip. It's a bit more complicated than that, and the find, sort, diff, grep, and cut are implemented in perl code. It also gets to use some assumptions I can make about our data, concerning file naming, dating, and sizing, and has no replacement for rsync's main magic, the incremental update of a file. Nonetheless, a similar approach might do well for you, as chances are, most of your changes are the addition and removal of files, with changes to existing files always entailing a change in size and/or timestamp.
Tim Conway [EMAIL PROTECTED] reorder name and reverse domain 303.682.4917 office, 303.921.0301 cell Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips, caesupport2 on AIM "There are some who call me.... Tim?" "Crowder, Mark" <[EMAIL PROTECTED]> Sent by: [EMAIL PROTECTED] 10/21/2002 08:37 AM To: [EMAIL PROTECTED] cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject: Any work-around for very large number of files yet? Classification: Yes, I've read the FAQ, just hoping for a boon... I'm in the process of relocating a large amount of data from one nfs server to another (Network Appliance filers). The process I've been using is to nfs mount both source and destination to a server (solaris8) and simply use rsync -a /source/ /dest . It works great except for the few that have > 10 million files. On these I get the following: ERROR: out of memory in make_file rsync error: error allocating core memory buffers (code 22) at util.c(232) It takes days to resync these after the cutover with tar, rather than the few hours it would take with rsync -- this is making for some angry users. If anyone has a work-around, I'd very much appreciate it. Thanks, Mark Crowder Texas Instruments, KFAB Computer Engineering email: [EMAIL PROTECTED] -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html