Re: Aw: Re: rsync 3.0.9 segmentation fault
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Using --no-inc-recursive would increase memory usage not reduce it. That option forces rsync to keep the entire tree in memory. On 03/19/2015 03:10 PM, devz...@web.de wrote: Hi Aron, i hope it`s ok for you if i bring this back on-list. Your issue and the way or possible fix to resolve it may be interesting for others too (that may include future searches etc) so with 3.1.1 we are a step further i don`t really have a clue what`s happening here but my next step would be taking a closer look on how the memory usage of rsync on the client and server grows. you could log it like this: while true;do ps -eo vsz,rss,sz,rsync|grep cron;sleep 10;done logfile does it grow continuously? does the oom situation reproducibly happen at a certain size ? what`s the client and server platform? how many files? (- https://rsync.samba.org/FAQ.html#5 ! ) regards roland *Gesendet:* Donnerstag, 19. März 2015 um 12:24 Uhr *Von:* Aron Rotteveel rotteveel.a...@gmail.com *An:* devz...@web.de *Betreff:* Re: rsync 3.0.9 segmentation fault In addition to my last message: * Client (sender) has 16GB's or RAM, of which only 6.5GB is used during peak. * I tried using --no-inc-recursive, but it does not solve the issue. What currrently is puzzling me is the question of why I am receiving these errors when my server seems to have plenty of memory to spare. -- Best regards / Met vriendelijke groet, Aron Rotteveel 2015-03-19 11:52 GMT+01:00 Aron Rotteveel rotteveel.a...@gmail.com: Hi Roland, I just upgrade both the client and host to 3.1.1 and seem to memory related issues now: ERROR: out of memory in make_file [sender] rsync error: error allocating core memory buffers (code 22) at util2.c(102) [sender=3.1.1] [sender] _exit_cleanup(code=22, file=util2.c, line=102): about to call exit(22) [Receiver] _exit_cleanup(code=22, file=io.c, line=1633): about to call exit(22) rsnapshot encountered an error! The program was invoked with these options: /usr/bin/rsnapshot -c \ /home/remotebackup/hosts/redacted/rsnapshot.conf sync -- Best regards / Met vriendelijke groet, Aron Rotteveel 2015-03-18 23:43 GMT+01:00 devz...@web.de: Hi, rsync 3.0.9 is quite ancient, more than 3 years old. A lot of bugs have been fixed since then. Is there a chance to update to the latest rsync version and retry with that ? regards Roland *Gesendet:* Dienstag, 17. März 2015 um 11:51 Uhr *Von:* Aron Rotteveel rotteveel.a...@gmail.com *An:* rsync@lists.samba.org *Betreff:* rsync 3.0.9 segmentation fault Hi, I am experiencing segfaults when transferring files via rsync though sudo. Setup: - Backupserver initiates the rsync command with --delete -vvv --no-inc-recursive --numeric-ids --delete-excluded --relative --rsync-path=/home/backupuser/rsync-wrapper.sh - rsync-wrapper.sh (on the client) contains /usr/bin/sudo /usr/bin/rsync $@; - user backupuser has sudo access to the rsync command - Both host and client are running 3.0.9 The transfer starts and some files are actually transferred. Once a certain file is reached (plain PHP file, no special characters or any other peculiarities) it segfaults. rsync host output: [sender] make_file(redacted/libraries/phputf8/mbstring/strlen.php,*,2) rsync: connection unexpectedly closed (51261222 bytes received so far) [Receiver] rsync error: unexplained error (code 139) at io.c(605) [Receiver=3.0.9] [Receiver] _exit_cleanup(code=12, file=io.c, line=605): about to call exit(139) rsnapshot encountered an error! The program was invoked with these options: /usr/bin/rsnapshot -c \ /home/remotebackup/hosts/redacted/rsnapshot.conf sync ERROR: /usr/bin/rsync returned 139 while processing backupuser@redacted:/backup/ Client output when using gdb to debug the coredump: warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff015fd000 Core was generated by `/usr/bin/rsync --server --sender -vvvlogDtprRe.Lsf --numeric-ids . /backup'. Program terminated with signal 11, Segmentation fault. #0 0x0035cda7b441 in memcpy () from /lib64/libc.so.6 Any help would be greatly appreciated. Please let me know if additional info is required to properly debug this issue. -- Best regards / Met vriendelijke groet, Aron Rotteveel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - --
Aw: Re: rsync 3.0.9 segmentation fault
Hi Aron, i hope its ok for you if i bring this back on-list. Your issue and the way or possible fix to resolve it may be interesting for others too (that may include future searches etc) so with 3.1.1 we are a step further i dont really have a clue whats happening here but my next step would be taking a closer look on how the memory usage of rsync on the client and server grows. you could log it like this: while true;do ps -eo vsz,rss,sz,rsyncgrep cron;sleep 10;done logfile does it grow continuously? does the oom situation reproducibly happen at a certain size ? whats the client and server platform? how many files? (- https://rsync.samba.org/FAQ.html#5 ! ) regards roland Gesendet:Donnerstag, 19. Mrz 2015 um 12:24 Uhr Von:Aron Rotteveel rotteveel.a...@gmail.com An:devz...@web.de Betreff:Re: rsync 3.0.9 segmentation fault In addition to my last message: Client (sender) has 16GBs or RAM, of which only 6.5GB is used during peak. I tried using --no-inc-recursive, but it does not solve the issue. What currrently is puzzling me is the question of why I am receiving these errors when my server seems to have plenty of memory to spare. -- Best regards / Met vriendelijke groet, Aron Rotteveel 2015-03-19 11:52 GMT+01:00 Aron Rotteveel rotteveel.a...@gmail.com: Hi Roland, I just upgrade both the client and host to 3.1.1 and seem to memory related issues now: ERROR: out of memory in make_file [sender] rsync error: error allocating core memory buffers (code 22) at util2.c(102) [sender=3.1.1] [sender] _exit_cleanup(code=22, file=util2.c, line=102): about to call exit(22) [Receiver] _exit_cleanup(code=22, file=io.c, line=1633): about to call exit(22) rsnapshot encountered an error! The program was invoked with these options: /usr/bin/rsnapshot -c /home/remotebackup/hosts/redacted/rsnapshot.conf sync -- Best regards / Met vriendelijke groet, Aron Rotteveel 2015-03-18 23:43 GMT+01:00 devz...@web.de: Hi, rsync 3.0.9 is quite ancient, more than 3 years old. A lot of bugs have been fixed since then. Is there a chance to update to the latest rsync version and retry with that ? regards Roland Gesendet:Dienstag, 17. Mrz 2015 um 11:51 Uhr Von:Aron Rotteveel rotteveel.a...@gmail.com An:rsync@lists.samba.org Betreff:rsync 3.0.9 segmentation fault Hi, I am experiencing segfaults when transferring files via rsync though sudo. Setup: - Backupserver initiates the rsync command with--delete -vvv --no-inc-recursive --numeric-ids --delete-excluded --relative --rsync-path=/home/backupuser/rsync-wrapper.sh - rsync-wrapper.sh (on the client) contains/usr/bin/sudo /usr/bin/rsync @; - user backupuser has sudo access to the rsync command - Both host and client are running 3.0.9 The transfer starts and some files are actually transferred. Once a certain file is reached (plain PHP file, no special characters or any other peculiarities) it segfaults. rsync host output: [sender] make_file(redacted/libraries/phputf8/mbstring/strlen.php,*,2) rsync: connection unexpectedly closed (51261222 bytes received so far) [Receiver] rsync error: unexplained error (code 139) at io.c(605) [Receiver=3.0.9] [Receiver] _exit_cleanup(code=12, file=io.c, line=605): about to call exit(139) rsnapshot encountered an error! The program was invoked with these options: /usr/bin/rsnapshot -c /home/remotebackup/hosts/redacted/rsnapshot.conf sync ERROR: /usr/bin/rsync returned 139 while processing backupuser@redacted:/backup/ Client output when using gdb to debug the coredump: warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff015fd000 Core was generated by /usr/bin/rsync --server --sender -vvvlogDtprRe.Lsf --numeric-ids . /backup. Program terminated with signal 11, Segmentation fault. #0 0x0035cda7b441 in memcpy () from /lib64/libc.so.6 Any help would be greatly appreciated. Please let me know if additional info is required to properly debug this issue. -- Best regards / Met vriendelijke groet, Aron Rotteveel -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync Batch mode with multicast
the man pages indicate that batch mode can use multicast Multicast transport protocols can be used to transfer the batch update files in parallel to many hosts at once, instead of sending the same data to every host individually.” I can’t seem to find a reliable example anywhere about using batch mode with multicast. Anyone here has had any success with that? Thanks t-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
