Thu, 2 Aug 2007 16:52:53 +0200, paul wrote: > On Thu 02 Aug 2007, Sven wrote: > > > > I once tried to rsync around 100 GB (10 million files), but version > > 2.6.6 needed too much RAM and was too slow. > > I believe 2.6.7 had a number of memory-saving options; each subsequent > version will of course have other improvements. 2.6.9 is the current > official release. > > If those files don't have hardlinks (or you're not interested > in preserving hardlinks),
No hardlinks. > you can greatly improve performance > by trying the 3.0.0 cvs snapshot. Yes, the cvs version from 20070803T08 started very modestly in terms of RAM, but after some minutes: > rsync -a --progress -v /c/ /data/c (source is mounted via nfs) ... ces/002495.ces 105851 100% 859.88kB/s 0:00:00 (xfer#23, to-check=5762/13367) *** glibc detected *** rsync: munmap_chunk(): invalid pointer: 0x080a8cb0 *** ======= Backtrace: ========= /lib/libc.so.6[0x2a97e6e1] rsync[0x807c0f0] rsync[0x804b782] rsync[0x8056215] rsync[0x8056825] rsync[0x806053a] rsync[0x8060d89] rsync[0x8062108] rsync[0x807556f] rsync[0x80618c5] /lib/libc.so.6(__libc_start_main+0xdc)[0x2a92ff9c] rsync[0x804a6c1] ======= Memory map: ======== 08048000-08092000 r-xp 00000000 00:14 389384 /linux/rsync-cvs/bin/rsync 08092000-08094000 rw-p 0004a000 00:14 389384 /linux/rsync-cvs/bin/rsync 08094000-0866a000 rw-p 08094000 00:00 0 [heap] 2a7c8000-2a7d2000 r-xp 00000000 08:02 2232639 /lib/libgcc_s.so.1 2a7d2000-2a7d4000 rw-p 00009000 08:02 2232639 /lib/libgcc_s.so.1 2a7f8000-2a82d000 r--s 00000000 08:02 1577380 /var/run/nscd/passwd 2a84e000-2a890000 rw-p 2a84e000 00:00 0 2a890000-2a8c5000 r--s 00000000 08:02 1577476 /var/run/nscd/group 2a8e6000-2a919000 r--p 00000000 08:02 526055 /usr/lib/locale/en_US/LC_CTYPE 2a919000-2a91a000 rw-p 2a919000 00:00 0 2a91a000-2aa42000 r-xp 00000000 08:02 2232595 /lib/libc-2.5.so 2aa42000-2aa43000 r--p 00128000 08:02 2232595 /lib/libc-2.5.so 2aa43000-2aa45000 rw-p 00129000 08:02 2232595 /lib/libc-2.5.so 2aa45000-2aa48000 rw-p 2aa45000 00:00 0 2aa48000-2aa4f000 r-xp 00000000 08:02 480803 /usr/lib/libpopt.so.0.0.0 2aa4f000-2aa51000 rw-p 00006000 08:02 480803 /usr/lib/libpopt.so.0.0.0 2aa6e000-2aa75000 r--s 00000000 08:02 542502 /usr/lib/gconv/gconv-modules.cache 2aa75000-2aa76000 rw-p 2aa75000 00:00 0 2aa76000-2aa91000 r-xp 00000000 08:02 2232588 /lib/ld-2.5.so 2aa91000-2aa93000 rw-p 0001a000 08:02 2232588 /lib/ld-2.5.so ffd49000-ffd54000 rw-p ffd49000 00:00 0 [stack] ffffe000-fffff000 r-xp ffffe000 00:00 0 rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32) rsync: connection unexpectedly closed (2322 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.0cvs] [ERROR: 12] > Is this a known problem? Sven
pgpQQdRiTXCbS.pgp
Description: PGP signature
-- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html