Rsync backup script with snapshot
Hi guys, Many thanks with your help on getting my backup script working with relative paths and LVM snapshots. I have it working pretty well now, and I thought I would let everyone see the finished product. I have published it on my wiki at http://wiki.receptiveit.com.au/mediawiki/index.php/Linux:_Backup_Strategies There is still a few things that I need to tidy up with this script, but for the time being, I am pretty happy with it. If anyone has any criticism or suggestions, please feel fee. aF -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Behavioral differences between versions.
On Wed 21 Jul 2010, Jeff Cleverley wrote: 1. When using the -z option, does rsync actually use /usr/bin/compress or does it use gzip? The man pages say it compresses data but doesn't specify what it compresses with. It uses the gzip algorithm but implements it internally; it doesn't call any external program. That wouldn't be at all practical on a communication link which is effectively interactive (one side sends data, the other side reacts). 2. I was looking for detailed documentation on behavioral changes of arguments between rsync versions. I suggest examining the NEWS and OLDNEWS files in the rsync source. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Behavioral differences between versions.
Paul, Thanks for the information. I'll see what I can find in the NEWS and OLDNEWS. Hopefully that will give me what I'm looking for on most of the questions. Am I correct in my assumption that Redhat does not tweak the rsync it ships with its distribution? Thanks, Jeff On Thu, Jul 22, 2010 at 3:20 AM, Paul Slootman paul+rs...@wurtel.net wrote: On Wed 21 Jul 2010, Jeff Cleverley wrote: 1. When using the -z option, does rsync actually use /usr/bin/compress or does it use gzip? The man pages say it compresses data but doesn't specify what it compresses with. It uses the gzip algorithm but implements it internally; it doesn't call any external program. That wouldn't be at all practical on a communication link which is effectively interactive (one side sends data, the other side reacts). 2. I was looking for detailed documentation on behavioral changes of arguments between rsync versions. I suggest examining the NEWS and OLDNEWS files in the rsync source. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync algorithm
Check file size makes sense, but how rsync checks times? If a file is copied from one side to another remote side, the time will be different, right? Sorry wrong question, the copy file should be able to reserve the mtime. If the rsync --times option is used then rsync will attempt to preserve modification times. You version of rsync may also have support of the --crtimes option. If this option is used then rsync will attempt to preserve create times. -- This email is protected by LBackup http://www.lbackup.org -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Problem backing up read-only files
I have a PC on which I run Windows XP and Cygwin 1.5. To this is attached an HP Media Vault mv2120 via Ethernet through a switch. I can copy files back and forth between the XP machine and the Media Vault, so that much seems to work fine. I would like to use rsync to back up some of the files on the XP machine to the Media Vault. This seems to work as it should as long as the files have write permission, but rsync fails to copy files that have only read permission. To demonstrate this, I created an 'rstest' directory under my Cygwin home directory and created in it two files, one with default permissions and the other without write permissions. bash-3.2$ mkdir rstest bash-3.2$ cd rstest bash-3.2$ echo hello, world rstest1.txt bash-3.2$ echo hello, world rstest2.txt bash-3.2$ chmod -w rstest1.txt bash-3.2$ ll total 2 -r--r--r-- 1 Gary Johnson None 13 Jul 22 17:02 rstest1.txt -rw-r--r-- 1 Gary Johnson None 13 Jul 22 17:02 rstest2.txt On the Media Vault, I created another 'rstest' directory. bash-3.2$ mkdir //hpmediavault/Documents/rstest bash-3.2$ ll -d //hpmediavault/Documents/rstest drwxr-xr-x 1 Gary Johnson None 0 Jul 22 17:04 //hpmediavault/Documents/rstest Then I attempted to transfer both files to the 'rstest' directory on the Media Vault. bash-3.2$ rsync -avv * //hpmediavault/Documents/rstest sending incremental file list delta-transmission disabled for local transfer or --whole-file rstest1.txt rstest2.txt rsync: rename /hpmediavault/Documents/rstest/.rstest1.txt.uCmUNq - rstest1.txt: Permission denied (13) total: matches=0 hash_hits=0 false_alarms=0 data=26 sent 188 bytes received 50 bytes 476.00 bytes/sec total size is 26 speedup is 0.11 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/main.c(1040) [sender=3.0.4] bash-3.2$ ll //hpmediavault/Documents/rstest total 1024 -rw-r--r-- 1 Gary Johnson None 13 Jul 22 17:02 rstest2.txt I thought that I might be able to work around the problem by omitting the -a flag, which implies -p, but the results were exactly the same. I am able to copy read-only files and preserve their permissions by using tar cf and tar xf in a pipeline, and that works fine for an initial backup, but for subsequent backups I really need a way to copy only those files that have changed. What can I do to fix or work around this problem? The version of rsync I'm using on Cygwin is rsync version 3.0.4 protocol version 30 Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, no IPv6, batchfiles, inplace, append, ACLs, no xattrs, iconv, symtimes Regards, Gary -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem backing up read-only files
Hi Gary, It looks like the problem occurs after rsync has transferred the file and is renaming the temporary file to the actual name. I am guessing that the HP media vault has Samba running on it, and it is actually denying access. As a long show, you could try --inplace, which will mean rsync does not need to rename the file. aF On 23/07/2010, at 11:04 AM, Gary Johnson wrote: I have a PC on which I run Windows XP and Cygwin 1.5. To this is attached an HP Media Vault mv2120 via Ethernet through a switch. I can copy files back and forth between the XP machine and the Media Vault, so that much seems to work fine. I would like to use rsync to back up some of the files on the XP machine to the Media Vault. This seems to work as it should as long as the files have write permission, but rsync fails to copy files that have only read permission. To demonstrate this, I created an 'rstest' directory under my Cygwin home directory and created in it two files, one with default permissions and the other without write permissions. bash-3.2$ mkdir rstest bash-3.2$ cd rstest bash-3.2$ echo hello, world rstest1.txt bash-3.2$ echo hello, world rstest2.txt bash-3.2$ chmod -w rstest1.txt bash-3.2$ ll total 2 -r--r--r-- 1 Gary Johnson None 13 Jul 22 17:02 rstest1.txt -rw-r--r-- 1 Gary Johnson None 13 Jul 22 17:02 rstest2.txt On the Media Vault, I created another 'rstest' directory. bash-3.2$ mkdir //hpmediavault/Documents/rstest bash-3.2$ ll -d //hpmediavault/Documents/rstest drwxr-xr-x 1 Gary Johnson None 0 Jul 22 17:04 //hpmediavault/Documents/rstest Then I attempted to transfer both files to the 'rstest' directory on the Media Vault. bash-3.2$ rsync -avv * //hpmediavault/Documents/rstest sending incremental file list delta-transmission disabled for local transfer or --whole-file rstest1.txt rstest2.txt rsync: rename /hpmediavault/Documents/rstest/.rstest1.txt.uCmUNq - rstest1.txt: Permission denied (13) total: matches=0 hash_hits=0 false_alarms=0 data=26 sent 188 bytes received 50 bytes 476.00 bytes/sec total size is 26 speedup is 0.11 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/main.c(1040) [sender=3.0.4] bash-3.2$ ll //hpmediavault/Documents/rstest total 1024 -rw-r--r-- 1 Gary Johnson None 13 Jul 22 17:02 rstest2.txt I thought that I might be able to work around the problem by omitting the -a flag, which implies -p, but the results were exactly the same. I am able to copy read-only files and preserve their permissions by using tar cf and tar xf in a pipeline, and that works fine for an initial backup, but for subsequent backups I really need a way to copy only those files that have changed. What can I do to fix or work around this problem? The version of rsync I'm using on Cygwin is rsync version 3.0.4 protocol version 30 Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, no IPv6, batchfiles, inplace, append, ACLs, no xattrs, iconv, symtimes Regards, Gary -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Behavioral differences between versions.
On Thu, 2010-07-22 at 12:42 -0600, Jeff Cleverley wrote: Am I correct in my assumption that Redhat does not tweak the rsync it ships with its distribution? The way to find out is to ask Red Hat or look in their SRPMs. Choose either client or server (they seem to be the same in this case--are they always?): http://ftp.redhat.com/pub/redhat/linux/enterprise/5Client/en/os/SRPMS/rsync-2.6.8-3.1.src.rpm http://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/SRPMS/rsync-2.6.8-3.1.src.rpm It looks like there's one bug fix related to memory allocation for extended attributes, but that's all. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem backing up read-only files
On 2010-07-23, Alex Ferrara wrote: Hi Gary, It looks like the problem occurs after rsync has transferred the file and is renaming the temporary file to the actual name. I am guessing that the HP media vault has Samba running on it, and it is actually denying access. As a long show, you could try --inplace, which will mean rsync does not need to rename the file. That worked for the initial transfer of that read-only file, but nebash-3.2$ touch * bash-3.2$ ll total 2 -r--r--r-- 1 Gary Johnson None 13 Jul 22 19:46 rstest1.txt -rw-r--r-- 1 Gary Johnson None 13 Jul 22 19:46 rstest2.txt bash-3.2$ rsync -vv --inplace * //hpmediavault/Documents/rstest delta-transmission disabled for local transfer or --whole-file rstest1.txt rstest2.txt rsync: open /hpmediavault/Documents/rstest/rstest1.txt failed: Permission denied (13) total: matches=0 hash_hits=0 false_alarms=0 data=26 sent 159 bytes received 50 bytes 418.00 bytes/sec total size is 26 speedup is 0.12 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/main.c(1040) [sender=3.0.4] bash-3.2$ ll //hpmediavault/Documents/rstest total 2048 -r 1 Gary Johnson None 13 Jul 22 19:42 rstest1.txt -rw--- 1 Gary Johnson None 13 Jul 22 19:46 rstest2.txt I guess that's to be expected since the man page says, in the description of the --inplace option, (4) a file that does not have write permissions can not be updated, That will let me do the initial backup of several more files that happen to be read-only, but unfortunately doesn't solve the problem in the long term. Thanks for your help. Regards, Gary -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem backing up read-only files
On Thu, 2010-07-22 at 19:57 -0700, Gary Johnson wrote: On 2010-07-23, Alex Ferrara wrote: Hi Gary, It looks like the problem occurs after rsync has transferred the file and is renaming the temporary file to the actual name. I am guessing that the HP media vault has Samba running on it, and it is actually denying access. As a long show, you could try --inplace, which will mean rsync does not need to rename the file. That worked for the initial transfer of that read-only file, but ne bash-3.2$ touch * bash-3.2$ ll total 2 -r--r--r-- 1 Gary Johnson None 13 Jul 22 19:46 rstest1.txt -rw-r--r-- 1 Gary Johnson None 13 Jul 22 19:46 rstest2.txt bash-3.2$ rsync -vv --inplace * //hpmediavault/Documents/rstest delta-transmission disabled for local transfer or --whole-file rstest1.txt rstest2.txt rsync: open /hpmediavault/Documents/rstest/rstest1.txt failed: Permission denied (13) total: matches=0 hash_hits=0 false_alarms=0 data=26 sent 159 bytes received 50 bytes 418.00 bytes/sec total size is 26 speedup is 0.12 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at /home/lapo/packaging/rsync-3.0.4-1/src/rsync-3.0.4/main.c(1040) [sender=3.0.4] bash-3.2$ ll //hpmediavault/Documents/rstest total 2048 -r 1 Gary Johnson None 13 Jul 22 19:42 rstest1.txt -rw--- 1 Gary Johnson None 13 Jul 22 19:46 rstest2.txt I guess that's to be expected since the man page says, in the description of the --inplace option, (4) a file that does not have write permissions can not be updated, That will let me do the initial backup of several more files that happen to be read-only, but unfortunately doesn't solve the problem in the long term. Try --chmod=u+w . One can generally copy read-only files to a unix filesystem with no trouble, because on a unix filesystem, renames check only the directory permissions, not the file permissions. (As for read-only directories, rsync is smart enough to temporarily enable user-write permission while it processes them.) But with a non-conformant destination filesystem, it looks like --chmod=u+w is the only thing that will solve your issue. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Behavioral differences between versions.
Matt, Thanks for the information. I've found Red Hat often slow and difficult to get an answer from. I'll check the differences between the rhel4 and rhel5 versions in the SRPMs. I'll also check the bug tracking. I have found one issue I thought was probably just a change in behavior that was reported as a bug. Jeff On Thu, Jul 22, 2010 at 8:34 PM, Matt McCutchen m...@mattmccutchen.net wrote: On Thu, 2010-07-22 at 12:42 -0600, Jeff Cleverley wrote: Am I correct in my assumption that Redhat does not tweak the rsync it ships with its distribution? The way to find out is to ask Red Hat or look in their SRPMs. Choose either client or server (they seem to be the same in this case--are they always?): http://ftp.redhat.com/pub/redhat/linux/enterprise/5Client/en/os/SRPMS/rsync-2.6.8-3.1.src.rpm http://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/SRPMS/rsync-2.6.8-3.1.src.rpm It looks like there's one bug fix related to memory allocation for extended attributes, but that's all. -- Matt -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Behavioral differences between versions.
On Thu, 2010-07-22 at 21:50 -0600, Jeff Cleverley wrote: I've found Red Hat often slow and difficult to get an answer from. That's not our problem. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
