Re: memory usage in rsync 3.0.3 -- how much RAM should I have to transfer 13 million files?

2008-08-15 Thread Aleksey Tsalolikhin
To answer my own question, my rsync process has grown to 1 Gig resident in memory and seems to be holding steady at size as it keeps chugging along. Boy am I glad incremental recursion has been added - thanks!! I would never have been able to do this rsync otherwise. Aleksey On 8/14/08,

Re: memory usage in rsync 3.0.3 -- how much RAM should I have to transfer 13 million files?

2008-08-14 Thread Aleksey Tsalolikhin
Hi. Let's say I have 10,000 files per directory. If I understood Wayne, rsync builds a list for all the files in the current dir, plus another list for the directory being read-ahead. So how much memory should rsync use, for 20,000 files? I did double-check with -v, and it says receiving

Re: memory usage in rsync 3.0.3 -- how much RAM should I have to transfer 13 million files?

2008-08-12 Thread Shachar Shemesh
Aleksey Tsalolikhin wrote: I've upgraded from rsync 2.6.9 to 3.0.3 on both ends, but memory usage is still too high. Why should rsync 3's memory usage depend on the number of files? Does it keep files it already knows should not be transferred in memory? If not, then maybe we should hold

Re: memory usage in rsync 3.0.3 -- how much RAM should I have to transfer 13 million files?

2008-08-12 Thread Wayne Davison
On Tue, Aug 12, 2008 at 11:46:11AM -0700, Aleksey Tsalolikhin wrote: How much memory will rsync use? I didn't specificy any of the switches that disable incremental recursion. It depends on your options, and possibly on the maximum number of files in a directory. I've seen a recursive scan

Re: memory usage in rsync 3.0.3 -- how much RAM should I have to transfer 13 million files?

2008-08-12 Thread Aleksey Tsalolikhin
Thank you, Wayne. My options are: --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive . / We don't have hard links, AFAIK. I am archiving 2 months of data, and then I will trying doing another rsync run, and I'll add the -v