Re: W: best way to clone server data using rsync
On Tue, May 01, 2012 at 11:38:26AM -0500, Indulekha wrote: Often I use something like: rsync --archive --one-file-system --hard-links --human-readable --inplace --numeric-ids --delete ... to duplicate a system, with the '~/.rsync-exclude' file containing ^ Is that a mistake? or am I musunderstanding the --delete option. Note: I am not familiar with rsync, but I am familiar with the term duplicate -- Religion is excellent stuff for keeping common people quiet. -- Napoleon Bonaparte -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120502061658.GC13218@tal
Re: W: best way to clone server data using rsync
On Wed, May 02, 2012 at 06:16:59PM +1200, Chris Bannister wrote: On Tue, May 01, 2012 at 11:38:26AM -0500, Indulekha wrote: Often I use something like: rsync --archive --one-file-system --hard-links --human-readable --inplace --numeric-ids --delete ... to duplicate a system, with the '~/.rsync-exclude' file containing ^ Is that a mistake? or am I musunderstanding the --delete option. Note: I am not familiar with rsync, but I am familiar with the term duplicate No mistake Chris, the --delete option deletes data from the copy that doesn't exist in the source. The idea is to have only the same data as the source you're copying, aside from the stuff to ignore listed in the ~/.rsync-exclude file (stuff like /etc/fstab, if you're making a bootable copy of your system on an external drive). The links I gave will give a lot more detail, and rsync's man page is quite good too, though perhaps a tad opaque in places for the uninitiated. :) -- ❤ ♫ ❤ ♫ ❤ ♫ ❤ Indulekha -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120502063806.GB28630@radhesyama
Re: W: best way to clone server data using rsync
On Tue, May 01, 2012 at 01:08:14PM +0200, Tuxoholic wrote: rsync seems like the right choice, but how will it handle job canceling when I'm done for the day? Yes. But, for the initial sync, I tend to prefer a tarpipe which is a lot quicker. ( cd /srcdir tar cf - . ) | ssh user@somehost '( cd /destdir tar xf - )' Add 'z' to the tar arguments (on both sides) if you have reasonable CPUs and your data is not already compressed (don't bother for MPEGs, JPEGs, etc.; or if one end of the transfer is an embedded system). Once that has run, or if it was interrupted, I finish off with an rsync just to be sure. (others have already commented on your rsync arguments.) -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120502091402.GG17757@debian
Re: W: best way to clone server data using rsync
On Tue, May 01, 2012 at 10:19:42PM +0200, Tuxoholic wrote: I noticed rsh is not installed, it's linked to /etc/alternatives/rsh, which is linked to /usr/bin/ssh. Calling rsh instead of ssh should avoid file encryption during transfer, at least that was the intention. That isn't what will happen. SSH will not allow you to turn off encryption. You may want to specify -O Ciphers=rc4 (or blowfish), both of which are quite fast. The socket options boost transfer speed quite a lot, I usually have 65355 buffers on my samba server, using rsync directly I can increase it some more. Have you tested this with ssh? It seems unlikely to me. -dsr- -- http://randomstring.org/~dsr/eula.html is hereby incorporated by reference. You can't fight for freedom by taking away rights. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120502135739.gy11...@randomstring.org
W: best way to clone server data using rsync
Hi list I'm about to clone 1TB of server data to a new file server I'm building. Cloning will happen over the internal network, and it will take several days, since I'll only run it while I'm awake/at home and checking up on the progress every once in a while. rsync seems like the right choice, but how will it handle job canceling when I'm done for the day? - will it resume files properly? - will it run some sort of check sum to verify file integrity, or will I have to run myself an integrity check like md5sum afterwards? What's the right set of parameters to call rsync from a shell script? I'm not used to sync such huge amount of files and directories, so I'm unclear about how to call rsync in this case. Here's what I got so far from google research: rsync --sockopts=SO_SNDBUF=128000,SO_RCVBUF=128000 -e rsh --archive \ --recursive --partial --partial-dir=rsync-part --progress --append \ --files-from=/root/LISTOFFILES.txt --log-file=/root/rsync.log \ root@myserver:/PATH2myOLDServerPool/* /mnt/Mount2myNewServerPool LISTOFFILES.txt was created using: rsync --list-only /root/LISTOFFILES.txt Can somebody comment on the parameters, e.g. how these will work fine with job canceling and how this will handle file integrity? Is it safe to hit ctr+c to cancel the job, or is a SIGHUP to the rsync task the recommended/necessary way? -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/blu0-smtp1800d3c7368bd6ddb7b248cd8...@phx.gbl
Re: W: best way to clone server data using rsync
On 5/1/12 2:08 PM, Tuxoholic wrote: [snip] - will it resume files properly? - will it run some sort of check sum to verify file integrity, or will I have to run myself an integrity check like md5sum afterwards? Yes it will resume and it will verify file integrity. Using the archive option like you've chosen below, it will make sure that the target is like the source. [snip] Is it safe to hit ctr+c to cancel the job, or is a SIGHUP to the rsync task the recommended/necessary way? It's safe to cancel, rsync will pick up what it has missed when it starts again. Regards, /Lars -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4f9fd24e.5000...@gmail.com
Re: W: best way to clone server data using rsync
On Tue, May 01, 2012 at 01:08:14PM +0200, Tuxoholic wrote: Hi list I'm about to clone 1TB of server data to a new file server I'm building. Cloning will happen over the internal network, and it will take several days, since I'll only run it while I'm awake/at home and checking up on the progress every once in a while. rsync seems like the right choice, but how will it handle job canceling when I'm done for the day? - will it resume files properly? I use rsync for syncing and backups everyday, but I always do babysit it or at least check it when it finishes. Not sure if it will resume files properly, but certainly it will update appropriately when used correctly. - will it run some sort of check sum to verify file integrity, or will I have to run myself an integrity check like md5sum afterwards? Certainly wouldn't hurt, though I don't think it's necessary... What's the right set of parameters to call rsync from a shell script? I'm not used to sync such huge amount of files and directories, so I'm unclear about how to call rsync in this case. Here's what I got so far from google research: rsync --sockopts=SO_SNDBUF=128000,SO_RCVBUF=128000 -e rsh --archive \ --recursive --partial --partial-dir=rsync-part --progress --append \ --files-from=/root/LISTOFFILES.txt --log-file=/root/rsync.log \ root@myserver:/PATH2myOLDServerPool/* /mnt/Mount2myNewServerPool LISTOFFILES.txt was created using: rsync --list-only /root/LISTOFFILES.txt Often I use something like: rsync --archive --one-file-system --hard-links --human-readable --inplace --numeric-ids --delete --progress --exclude-from '/home/indulekha/.rsync-exclude / /copy_of_system/ to duplicate a system, with the '~/.rsync-exclude' file containing files and directories to ignore. Hopefully it's obvious that copy_of_system is a path you determine. Can somebody comment on the parameters, e.g. how these will work fine with job canceling and how this will handle file integrity? https://www.linux.com/news/enterprise/storage/8200-back-up-like-an-expert-with-rsynchttp://www.thegeekstuff.com/2010/09/rsync-command-examples/ Is it safe to hit ctr+c to cancel the job, or is a SIGHUP to the rsync task the recommended/necessary way? ctrl-c will work. HTH! -- ❤ ♫ ❤ ♫ ❤ ♫ ❤ Indulekha -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120501163826.GB15438@radhesyama
Re: W: best way to clone server data using rsync
On Tue, May 01, 2012 at 01:08:14PM +0200, Tuxoholic wrote: Here's what I got so far from google research: rsync --sockopts=SO_SNDBUF=128000,SO_RCVBUF=128000 -e rsh --archive \ --recursive --partial --partial-dir=rsync-part --progress --append \ --files-from=/root/LISTOFFILES.txt --log-file=/root/rsync.log \ root@myserver:/PATH2myOLDServerPool/* /mnt/Mount2myNewServerPool You don't need the sockopts. -a includes recursive. You probably don't want partial or partial-dir. You probably don't want append. Having rsh installed is a bad security risk. Install ssh, make sure you can ssh from this box to myserver as root. Change -e rsh to -e ssh. -dsr- -- http://randomstring.org/~dsr/eula.html is hereby incorporated by reference. You can't fight for freedom by taking away rights. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120501164330.gx11...@randomstring.org
Re: W: best way to clone server data using rsync
I noticed rsh is not installed, it's linked to /etc/alternatives/rsh, which is linked to /usr/bin/ssh. Calling rsh instead of ssh should avoid file encryption during transfer, at least that was the intention. The socket options boost transfer speed quite a lot, I usually have 65355 buffers on my samba server, using rsync directly I can increase it some more. partial-dir does not work along with append, so I got rid of it. I'll leave the rest as is minus the recursive option Thanks for the inputs! On 01.05.2012 18:43, Dan Ritter wrote: On Tue, May 01, 2012 at 01:08:14PM +0200, Tuxoholic wrote: Here's what I got so far from google research: rsync --sockopts=SO_SNDBUF=128000,SO_RCVBUF=128000 -e rsh --archive \ --recursive --partial --partial-dir=rsync-part --progress --append \ --files-from=/root/LISTOFFILES.txt --log-file=/root/rsync.log \ root@myserver:/PATH2myOLDServerPool/* /mnt/Mount2myNewServerPool You don't need the sockopts. -a includes recursive. You probably don't want partial or partial-dir. You probably don't want append. Having rsh installed is a bad security risk. Install ssh, make sure you can ssh from this box to myserver as root. Change -e rsh to -e ssh. -dsr- -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/blu0-smtp363466e7fa2c0053691fdd4d8...@phx.gbl
