[BackupPC-users] rsync out of memory
Hi, I decided to implement BackupPC at a second location as it is really reliable and nearly perfect! ;) Unfortunately, I'm getting errors when backing up one host. My debugging gave the following results: bash-4.1$ ./BackupPC_dump -v -f pdc.evs-nb.de cmdSystemOrEval: about to system /bin/ping -c 1 -w 3 abc.domain.de [...] full backup started for directory / started full dump, share=/ Running: /usr/bin/ssh -q -x -l root abc.domain.de /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive -x -z -v --no-inc-recursive -v --ignore-times . / Xfer PIDs are now 9933 xferPids 9933 Got remote protocol 30 Negotiated protocol version 28 Remote[2]: [sender] expand file_list pointer array to 524288 bytes, did move Remote[2]: [sender] expand file_list pointer array to 1048576 bytes, did move Xfer PIDs are now 9933,9934 xferPids 9933,9934 create d 555 0/04096 . Out of memory! Parent read EOF from child: fatal error! Done: 0 files, 0 bytes Got fatal error during xfer (Child exited prematurely) [...] CheckHostAlive: returning 0.156 Backup aborted (Child exited prematurely) Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0) dump failed: Child exited prematurely I started rsync manually with the same options as above and I monitored the memory consumption on the sender. This time it succeeded: total: matches=26625 hash_hits=2561895 false_alarms=35 data=5775632621 sent 2144313 bytes received 3721731504 bytes 5878257.01 bytes/sec total size is 5845056098 speedup is 1.57 [root@backup srv]# rsync uses around 112M, which should not be too much. Especially as the host hast still approx 1G of free memory. top is saying: 3620 root 20 0 112m 8036 952 R 69.5 0.2 3:17.46 rsync As you can see, I already added the --no-inc-recursive (reduce memory usage) and the -x (do not cross file systems) option to the rsync command. Still no go. Rsync continues telling me about out of memory. I can't believe there should be too much files in it as I store all other data except the OS itself on different partitions. /dev/mapper/vg1_pdc-root 12G 5.5G 5.5G 50% / Any other ideas how I can backup this server? I'm still wondering as this server is not a very uncommon setup- just default OS. Anyone having a clue? Greetings Christian -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync out of memory
On Thu, Nov 20, 2014 at 2:07 PM, Christian Völker chrisc...@knebb.de wrote: Got remote protocol 30 Negotiated protocol version 28 Remote[2]: [sender] expand file_list pointer array to 524288 bytes, did move Remote[2]: [sender] expand file_list pointer array to 1048576 bytes, did move Xfer PIDs are now 9933,9934 xferPids 9933,9934 create d 555 0/04096 . Out of memory! Parent read EOF from child: fatal error! I started rsync manually with the same options as above and I monitored the memory consumption on the sender. This time it succeeded: total: matches=26625 hash_hits=2561895 false_alarms=35 data=5775632621 Native rsync 3.x versions can use protocol 30 which can handle the directory tree incrementally. Backuppc's version uses protocol 28 (and forces the remote to use that also) which must xfer the entire tree first and hold it in RAM at both ends while they walk it doing the comparisions. sent 2144313 bytes received 3721731504 bytes 5878257.01 bytes/sec total size is 5845056098 speedup is 1.57 [root@backup srv]# rsync uses around 112M, which should not be too much. Especially as the host hast still approx 1G of free memory. top is saying: 3620 root 20 0 112m 8036 952 R 69.5 0.2 3:17.46 rsync I'm not actually sure from your error message which end was out of memory. How do things look on the target system while the backuppc backup runs? As you can see, I already added the --no-inc-recursive (reduce memory usage) and the -x (do not cross file systems) option to the rsync command. Still no go. Rsync continues telling me about out of memory. I can't believe there should be too much files in it as I store all other data except the OS itself on different partitions. /dev/mapper/vg1_pdc-root 12G 5.5G 5.5G 50% / Any other ideas how I can backup this server? I'm still wondering as this server is not a very uncommon setup- just default OS. Anyone having a clue? Memory use should relate to the number of files more than the total size. Do any directories have a huge number of tiny files? -- Les Mikesell lesmikes...@gmail.com -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync out of memory
Hi, Christian Völker wrote on 2014-11-20 21:07:59 +0100 [[BackupPC-users] rsync out of memory]: [...] Unfortunately, I'm getting errors when backing up one host. My debugging gave the following results: bash-4.1$ ./BackupPC_dump -v -f pdc.evs-nb.de cmdSystemOrEval: about to system /bin/ping -c 1 -w 3 abc.domain.de so ClientNameAlias for pdc.evs-nb.de is abc.domain.de? ;-) [...] Running: /usr/bin/ssh -q -x -l root abc.domain.de /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive -x -z -v --no-inc-recursive -v --ignore-times . / [...] Anyone having a clue? Yes. Remove the '-z' option. It won't work with File::RsyncP. Regards, Holger -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync out of memory
Hi Les, Native rsync 3.x versions can use protocol 30 which can handle the directory tree incrementally. Backuppc's version uses protocol 28 (and forces the remote to use that also) which must xfer the entire tree first and hold it in RAM at both ends while they walk it doing the comparisions. Is there a way to force protocol version 30 instead? Obviously with v28 the --no-incremental-recursive does not have any effect. I'm not actually sure from your error message which end was out of memory. How do things look on the target system while the backuppc backup runs? The host to be backed up seems to run out of memory. Has 4GB and while rsync runs the number of page cache increases so just around 100M remain as free memory. Though, rsync does not use more than 112M. Swapping apears but I was not able to catch the exact moment when the oom appears. Memory use should relate to the number of files more than the total size. Do any directories have a huge number of tiny files? Not really. See the output of find DIR -type f | wc -l for various dirs: bin: 86 dev: 43 lib: 2999 lib64: 281 media/: 0 opt/: 0 root/: 21 selinux/: 0 store/: 0 tmp/: 1 boot/: 22 etc/: 1222 mnt/: 37 proc/: 37101 sbin/: 190 srv/: 44 sys/: 6831 usr/: 54306 var: 7176 The huge directory is not to be backed up at this stage because it is mounted and rsync got the -x option. But for completenes: home: 57949 Any further ideas? Greetings Christian -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync out of memory
Hi so ClientNameAlias for pdc.evs-nb.de is abc.domain.de? ;-) You got me! Always the same with this security by obscurity thing! ;-) [...] Running: /usr/bin/ssh -q -x -l root abc.domain.de /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive -x -z -v --no-inc-recursive -v --ignore-times . / [...] Anyone having a clue? Yes. Remove the '-z' option. It won't work with File::RsyncP. Got it! Backup is running fine now. Strange. As I am backing up through a slow line is there another option to improve speed by using compression? I can obviously set the compression parameter for ssh- is it a good idea? Thanks for this great hint! Greetings Christian -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync out of memory
On Thu, Nov 20, 2014 at 3:04 PM, Christian Völker chrisc...@knebb.de wrote: Got it! Backup is running fine now. Strange. As I am backing up through a slow line is there another option to improve speed by using compression? I can obviously set the compression parameter for ssh- is it a good idea? I've never actually measured the difference, but I have added the -C option after $sshPath in the RsyncClientCmd for remote targets and it at least doesn't break anything. -- Les Mikesell lesmikes...@gmail.com -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=157005751iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync out of memory error...
Hi Les, Thanks for the response. This server is a 64bit server with 8gb of ram. Don't really see it running out of ram, but I didn't consider the processes. I'll check that! On 08/26/2013 09:01 AM, Gregory Malsack wrote: Hey All, Full backup worked fine, however incremental backup is throwing this error. Getting the same thing on my backup server and storage controller at another office as well. Exact same setup Aug 26 05:59:23 sc rsyncd[1357]: building file list Aug 26 05:59:24 sc rsyncd[1357]: ERROR: out of memory in receive_sums [sender] Aug 26 05:59:24 sc rsyncd[1357]: rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.6] Aug 26 05:59:24 sc xinetd[28196]: EXIT: rsync status=22 pid=1357 duration=2(sec) Any thoughts? -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58041391iu=/4140/ostg.clktrk___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync out of memory error...
On Thu, Sep 5, 2013 at 8:52 AM, Gregory Malsack gmals...@coastalacq.com wrote: Hi Les, Thanks for the response. This server is a 64bit server with 8gb of ram. Don't really see it running out of ram, but I didn't consider the processes. I'll check that! I've never seen error messages quite like that, but it looks like the error is coming from the sending side (and a standalone rsync started by xinetd?). Note that rsync loads the whole directory structure into RAM at both ends before they walk the list doing comparisions, so the amount of RAM needed will relate to the number of files under the starting point. If you have a reasonable division under a few top level directories, you could split them into separate 'shares'. Aug 26 05:59:23 sc rsyncd[1357]: building file list Aug 26 05:59:24 sc rsyncd[1357]: ERROR: out of memory in receive_sums [sender] Aug 26 05:59:24 sc rsyncd[1357]: rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.6] Aug 26 05:59:24 sc xinetd[28196]: EXIT: rsync status=22 pid=1357 duration=2(sec) -- Les Mikesell lesmikes...@gmail.com -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58041391iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] rsync out of memory error...
Hey All, Full backup worked fine, however incremental backup is throwing this error. Getting the same thing on my backup server and storage controller at another office as well. Exact same setup Aug 26 05:59:23 sc rsyncd[1357]: building file list Aug 26 05:59:24 sc rsyncd[1357]: ERROR: out of memory in receive_sums [sender] Aug 26 05:59:24 sc rsyncd[1357]: rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.6] Aug 26 05:59:24 sc xinetd[28196]: EXIT: rsync status=22 pid=1357 duration=2(sec) Any thoughts? -- Greg -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] rsync out of memory error...
On Mon, Aug 26, 2013 at 9:01 AM, Gregory Malsack gmals...@coastalacq.com wrote: Hey All, Full backup worked fine, however incremental backup is throwing this error. Getting the same thing on my backup server and storage controller at another office as well. Exact same setup Aug 26 05:59:23 sc rsyncd[1357]: building file list Aug 26 05:59:24 sc rsyncd[1357]: ERROR: out of memory in receive_sums [sender] Aug 26 05:59:24 sc rsyncd[1357]: rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.6] Aug 26 05:59:24 sc xinetd[28196]: EXIT: rsync status=22 pid=1357 duration=2(sec) Any thoughts? Wild guess - you need more memory But you could be hitting a process limit configured in the OS. -- Les Mikesell lesmikes...@gmail.com -- Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511iu=/4140/ostg.clktrk ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
