Les Mikesell wrote at about 12:16:49 -0500 on Monday, March 11, 2013: > On Mon, Mar 11, 2013 at 11:57 AM, <backu...@kosowsky.org> wrote: > > > > I still suggest trying a native rsync copy of the server to determine > > max memory usage. Then we will know if it's a BackupPC or an rsync > > (protocol <= 28) problem. If it's an rsync problem, buy more memory or > > split into smaller shares. If it's a backupppc problem, then fix the > > problem. > > I'm not sure how it helps to know if native rsync works. Perl's data > structures are bigger to permit its dynamic typing concepts and > reference counting so it will naturally need more memory.
It should still be proportional... Plus, the scale-up depends in large part how File::RsyncP is implemented... I believe some of it is implemented in C (hence, the need for compilation) and it is possible/likely that some of the data is stored in more efficient C-like data structures. Indeed, if File::RsyncP blindly used Perl memory storage, we *all* would have encountered memory issues on much smaller file sets. For example, on a 64 bit platform (not necessary 64 bit Perl), storing an integer takes 24 bytes, storing a string takes 48 bytes plus multiples of 8 bytes to store the contents. Hashes, take 150-200 bytes just for each hash key, not including the memory required to store the hash value. As they say, "to estimate memory usage of Perl, assume a reasonable algorithm for memory allocation, multiply that estimate by 10, and while you still may miss the mark, at least you won’t be quite so astonished." Given that I can back up shares containing 500,000+ files on a plugcomputer with only 512MB of memory with no swap and still have memory to spare for 2-3 simultaneous backups, I think it's likely that File::RsyncP is not just blindly using inefficient perl storage... ------------------------------------------------------------------------------ Symantec Endpoint Protection 12 positioned as A LEADER in The Forrester Wave(TM): Endpoint Security, Q1 2013 and "remains a good choice" in the endpoint security space. For insight on selecting the right partner to tackle endpoint security challenges, access the full report. http://p.sf.net/sfu/symantec-dev2dev _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
