Re: Yet another rsync incremental thread
OK, so here I'm giving you the "final" version of the script timestamp.sh working for me: - #!/bin/tcsh #Turn on debug info set DEBUG = 1 #Name of the backup mount (partition) set MOUNTNAME = data #Name of the backup directory set BACKUP = backup if( $DEBUG ) then echo "Starting script at `date +"%Y-%m-%d-%H%M%S"`" endif #Check if we have the good arg number if( $# > 0 ) then if( $DEBUG ) then echo "There are arguments" endif if( $# < 2 ) then if( $DEBUG ) then echo "There is one argument, setting it as host name" endif set RSYNC_HOST_NAME = $1 else echo "Too many arguments.\ Usage: $0 or\ $0 - without arguments, in which case\ RSYNC_HOST_NAME environment variable must indicate a host name." exit endif else if( ! $?RSYNC_HOST_NAME ) then echo "No RSYNC_HOST_NAME detected.\ Usage: $0 or\ $0 - without arguments, in which case RSYNC_HOST_NAME\ environment variable must indicate a host name." exit endif if( $DEBUG ) then echo "RSYNC_HOST_NAME = $RSYNC_HOST_NAME" endif #Change to working directory cd /mnt/$MOUNTNAME/$RSYNC_HOST_NAME if( $status ) then echo "Directory /mnt/$MOUNTNAME/$RSYNC_HOST_NAME does not exist" exit endif #Check if the backup directory exists if( ! ( -e $BACKUP ) ) then echo "Directory $BACKUP is missing in $PWD" exit endif #Set up useful variables set DATE = `date +"%Y-%m-%d-%H%M%S"` set AVAIL = `df | grep $MOUNTNAME | awk -F' ' '{print $4}'` set SIZE = `du -s $BACKUP | awk -F' ' '{print $1}'` if ( $DEBUG ) then echo "DATE = $DATE \ AVAIL = $AVAIL \ SIZE = $SIZE" endif if( ! ( -e lastest ) ) then if( $DEBUG) then echo "Moving backup and creating lastest." endif mv -f $BACKUP $DATE ln -sf $DATE lastest else if( $DEBUG ) then echo "Moving backup and lastest." endif mv -f $BACKUP $DATE rm -f lastest ln -sf $DATE lastest endif while ( $AVAIL < $SIZE ) if ( $DEBUG ) then echo "Removing older backup." endif rm -Rf `ls -1 | grep $1 | head -n 1` end unset * echo "Rotation successful." - with this rsync.conf: - syslog facility = local4 list = no port = 873 pid file = /var/run/rsyncd.pid uid = rsync [serveuranm] comment = Sauvegarde du serveur ANM path = /mnt/data/serveuranm.domaineanm.fr/ list = true max connections = 0 read only = false uid = serveuranm gid = rsync post-xfer exec = /mnt/data/timestamp.sh > /mnt/data/timestamp.log 2>&1 - I've putted final into quotes because this is working for my initial server, but I already have in mind some changes: - Convert the script to sh for better portability and also because tcsh scripting seems odd ( http://www.grymoire.com/Unix/CshTop10.txt ) - Make the script more generic - Make the script multi-modules (mainly for deleting content) I'm not sure if here's the right place to share it. Feel free to comment my work and please let me know if you use it as is or as a base for another script. Best regards, Vitorio -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another rsync incremental thread
I've made a simpler test: freenas:~# cat /var/etc/rsyncd.conf syslog facility = local4 list = no port = 873 pid file = /var/run/rsyncd.pid uid = rsync [serveuranm] comment = Sauvegarde du serveur ANM path = /mnt/data/serveuranm/ list = true max connections = 0 read only = false uid = serveuranm gid = rsync post-xfer exec = echo RSYNC_HOST_NAME >& /var/log/timestamp.log With this, I should see RSYNC_HOST_NAME on my timestamp.log after rsync runs, no? But nothing happens. My log file still empty... But with post-xfer exec = echo RSYNC_HOST_NAME > /var/log/timestamp.log It works! When I run rsync, RSYNC_HOST_NAME appears on timestamp.log! So I deduct that rsync calls a sh shell and not tcsh (I only have those on my box). Then I tried with post-xfer exec = /mnt/data/timestamp.sh $RSYNC_HOST_NAME > /var/log/ timestamp.log 2 >&1 It calls my script but without argument as I receive an error from my script in the log: Usage: /mnt/data/timestamp.sh But it's OK, from here I know what to do by myself. Thanks Matt, as always pointing on the good direction. When I'll be big, I wanna be like you ;) Best regards, Vitorio -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another rsync incremental thread
Hello there! Le 28 juil. 08 à 16:24, Matt McCutchen a écrit : (This original message doesn't seem to have gone to the list.) Actually it is here http://www.mail-archive.com/rsync@lists.samba.org/msg22126.html , but I messed around mail replies. Dunno exactly what I did... Doesn't matter. My guess is that the script is running but failing, and silently because the rsync daemon redirects its stdout and stderr to /dev/null. One of the first things a pre-xfer or post-xfer script should do is redirect its stdout and stderr to a log file somewhere that you can review in case of problems. In bash, the code for that is: exec >>/PATH/TO/LOG 2>&1 (I don't know tcsh.) I've tried this pertinent suggestion. Unfortunately FreeNAS doesn't have bash built-in and exec works a little bit different in tcsh. I found how to have a log of it like this: freenas:/mnt/data/serveuranm# cat /var/etc/rsyncd.conf syslog facility = local4 list = no port = 873 pid file = /var/run/rsyncd.pid uid = rsync [serveuranm] comment = Sauvegarde du serveur ANM path = /mnt/data/serveuranm/ list = true max connections = 0 read only = false uid = serveuranm gid = rsync post-xfer exec = /mnt/data/timestamp.sh $RSYNC_HOST_NAME >& /var/log/ timestamp.log >& in tcsh redirects stdout and stderr, like cmd > file 2>&1 in bash. I've also tested launching the script with this exact command (just replacing the $RSYNC_HOST_NAME, of course). It runs fine and logs activity to /var/log/timestamp.log as expected And why the failure? Your script seems to assume a particular working directory, because it goes looking for "data" and "backup" in that directory. Note that it will inherit the working directory in which "rsync --daemon" was run to start the daemon. If that isn't what your script wants, add an appropriate "cd" command. Good remark as well. I've changed the script to cd to the right directory before executing commands. Current version looks like this: freenas:/mnt/data/serveuranm# cat /mnt/data/timestamp.sh #!/bin/tcsh #exec $0 >& /var/log/timestamp.log #Turn on debug info set DEBUG = 0 #Check if we have the good arg number if( $# < 1 || $# > 1 ) then echo "Usage: $0 " exit endif #Name of the backup mount (partition) set MOUNTNAME = data #Change to working directory cd /mnt/$MOUNTNAME/$1 if( $status ) then echo "Directory /mnt/$MOUNTNAME/$1 does not exist" exit endif echo $PWD #cd /mnt/$MOUNTNAME/$1 #echo $PWD #Name of the backup directory set BACKUP = backup #Check if the backup directory exists if( ! ( -e $BACKUP ) ) then echo "Directory backup is missing" exit endif #Set up useful variables set DATE = `date +"%Y-%m-%d-%H%M%S"` set AVAIL = `df | grep $MOUNTNAME | awk -F' ' '{print $4}'` set SIZE = `du -s $BACKUP | awk -F' ' '{print $1}'` if ( $DEBUG ) then echo "DATE = $DATE \ AVAIL = $AVAIL \ SIZE = $SIZE" endif if( ! ( -e lastest ) ) then mv -f $BACKUP lastest else mv -f lastest $1$DATE mv -f $BACKUP lastest endif while ( $AVAIL < $SIZE ) rm -Rf `ls -1 | grep $1 | head -n 1` end unset * echo "Rotation successful" freenas:/mnt/data/serveuranm# Note that I forced it to echo something if it runs, in all cases. I also tested the changes on the script running it alone. All works fine when running from any directory. With all those modifications, the result is always the same. The script isn't called. I can say it for sure now, log file still remain empty. I also checked permission on the log file. Everyone have access to / var/log directory and timestamp.log is 666 so writable by anyone. Do you already used post-xfer with plain rsync? I don't know why, I think this is why it's not working. Or maybe I should specify explicitly the user in rsync command in cygwin? I don't really think so, because client size fullfil it's policy. It opens the connection with the server, push all the content and close connection like a charm. If there was an username pb, it wouldn't even transfer files, would it? Thanks indeed for the help, Vitorio-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another rsync incremental thread
> Le 28 juil. 08 à 09:39, macuserfr a écrit : > > Hi folks, that's me again... > > > > Well, following Matt's suggestion, I'm trying to setup my server to > > call a post transfer script. Guess what? It's not working and I > > don't know why. I've looked around other posts in this mail list and > > googling around. Didn't found what's wrong with my setup. So rsync > > runs nicely but don't call the post-transfer script. The post > > transfer script runs fine when I run it manually with root, rsync > > user or the module defined user (serveuranm). No errors in client > > side nor in rsyncd log. There is maybe a path/user mistake somewhere. (This original message doesn't seem to have gone to the list.) My guess is that the script is running but failing, and silently because the rsync daemon redirects its stdout and stderr to /dev/null. One of the first things a pre-xfer or post-xfer script should do is redirect its stdout and stderr to a log file somewhere that you can review in case of problems. In bash, the code for that is: exec >>/PATH/TO/LOG 2>&1 (I don't know tcsh.) And why the failure? Your script seems to assume a particular working directory, because it goes looking for "data" and "backup" in that directory. Note that it will inherit the working directory in which "rsync --daemon" was run to start the daemon. If that isn't what your script wants, add an appropriate "cd" command. Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another rsync incremental thread
Hmmm, I also tried this to use variable from the script calling instead of from the script body, same result (script doesn't run : / ) : freenas:/mnt/data/serveuranm# cat ../timestamp.sh #!/bin/tcsh #Turn on debug info set DEBUG = 0 #Check if we have the good arg number if( $# < 1 || $# > 1 ) then echo "Usage: $0 " exit endif #Name of the backup directory set BACKUP = backup #Name of the backup mount (partition) set MOUNTNAME = data #Check if the backup directory exists if( ! ( -e $BACKUP ) ) then echo "Directory backup is missing" exit endif #Set up useful variables set DATE = `date +"%Y-%m-%d-%H%M%S"` set AVAIL = `df | grep $MOUNTNAME | awk -F' ' '{print $4}'` set SIZE = `du -s $BACKUP/ | awk -F' ' '{print $1}'` if ( $DEBUG ) then echo "DATE = $DATE \ AVAIL = $AVAIL \ SIZE = $SIZE" endif if( ! ( -e lastest ) ) then mv -f $BACKUP lastest else mv -f lastest $1$DATE mv -f $BACKUP lastest endif while ( $AVAIL < $SIZE ) rm -Rf `ls -1 | grep $1 | head -n 1` end unset * freenas:/mnt/data/serveuranm# cat /var/etc/rsyncd.conf syslog facility = local4 list = no port = 873 pid file = /var/run/rsyncd.pid uid = rsync [serveuranm] comment = Sauvegarde du serveur ANM path = /mnt/data/serveuranm/ list = true max connections = 0 read only = false uid = serveuranm gid = rsync post-xfer exec = /mnt/data/timestamp.sh $RSYNC_HOST_NAME Thanks for any clue of what's wrong. Best regards, Vitorio Le 28 juil. 08 à 09:39, macuserfr a écrit : Hi folks, that's me again... Well, following Matt's suggestion, I'm trying to setup my server to call a post transfer script. Guess what? It's not working and I don't know why. I've looked around other posts in this mail list and googling around. Didn't found what's wrong with my setup. So rsync runs nicely but don't call the post-transfer script. The post transfer script runs fine when I run it manually with root, rsync user or the module defined user (serveuranm). No errors in client side nor in rsyncd log. There is maybe a path/user mistake somewhere. Client side: cwrsync with rsync 3.0.2 and modded cygwin1.dll in order to have long name support. cwrsync.cmd batch rsync file content: @ECHO OFF REM * REM REM CWRSYNC.CMD - Batch file template to start your rsync command (s). REM REM By Tevfik K. (http://itefix.no) REM * REM Make environment variable changes local to this batch file SETLOCAL REM ** CUSTOMIZE ** Specify where to find rsync and related files (C: \CWRSYNC) SET CWRSYNCHOME=%PROGRAMFILES%\CWRSYNC REM Set CYGWIN variable to 'nontsec'. That makes sure that permissions REM on your windows machine are not updated as a side effect of cygwin REM operations. SET CYGWIN=nontsec nodosfilewarning codepage:utf8 SET LC_CTYPE="C-UTF-8" REM Set HOME variable to your windows home directory. That makes sure REM that ssh command creates known_hosts in a directory you have access. SET HOME=%HOMEDRIVE%%HOMEPATH% REM Make cwRsync home as a part of system PATH to find required DLLs SET CWOLDPATH=%PATH% SET PATH=%CWRSYNCHOME%\BIN;%PATH% REM Windows paths may contain a colon (:) as a part of drive designation and REM backslashes (example c:\, g:\). However, in rsync syntax, a colon in a REM path means searching for a remote host. Solution: use absolute path 'a la unix', REM replace backslashes (\) with slashes (/) and put -/cygdrive/- in front of the REM drive letter: REM REM Example : C:\WORK\* --> /cygdrive/c/work/* REM REM Example 1 - rsync recursively to a unix server with an openssh server : REM REM rsync -r /cygdrive/c/work/ remotehost:/home/user/work/ REM REM Example 2 - Local rsync recursively REM REM rsync -r /cygdrive/c/work/ /cygdrive/d/work/doc/ REM REM Example 3 - rsync to an rsync server recursively : REM(Double colons?? YES!!) REM REM rsync -r /cygdrive/c/doc/ remotehost::module/doc REM REM Rsync is a very powerful tool. Please look at documentation for other options. REM REM ** CUSTOMIZE ** Enter your rsync command(s) here rsync -ab --link-dest="/lastest" --size-only --chmod=ugo=rwX / cygdrive/c/DossiersPartages 172.20.30.194::serveuranm/backup Server side: freenas:~# cat /var/etc/rsyncd.conf syslog facility = local4 list = no port = 873 pid file = /var/run/rsyncd.pid uid = rsync [serveuranm] comment = Sauvegarde du serveur ANM path = /mnt/data/serveuranm/ list = true max connections = 0 read only = false uid = serveuranm gid = rsync post-xfer exec = /mnt/data/timestamp.sh freenas:~# cat /var/log/rsyncd.log Jul 24 09:46:13 freenas rsyncd[18257]: connect from serveuranm.domaineanm.fr (172.20.30.197) Jul 24 09:46:13 freenas rsyncd[18258]: rsync to serveuranm/backup from serveuranm.domaineanm.fr (172.20.30.197) Jul 24 09:46:13 freenas rsyncd[18258]: receiving file list Jul 24 10:57:40 freenas
Re: Yet another rsync incremental thread
Le 19 juil. 08 à 20:13, Matt McCutchen a écrit : On Sat, 2008-07-19 at 15:45 +0200, Mac User FR wrote: In another hand, don't you think we could add the content of this script (revised, of course) to make rsync a complete incremental backup solution without needing 3rd part software? Isn't a good idea? Why? The goal of rsync is to copy files, not to be a complete incremental backup solution by itself, even though several of its options are motivated by this use case. I think the current state of affairs with a wide variety of available rsync-based backup solutions (rsnapshot, dirvish, ccollect, various homegrown scripts including yours, ...) is just fine. Different users prefer different solutions, so I don't think arbitrarily choosing one to include in rsync would be appropriate. Matt OK, I though you though this way. I'm not wanting to impose my script particularly, just thinking about people with less computer skills than ours. Making things easier for them. This is not my case, I'm not afraid about coding some stuff like I did for Panther being able to save resource forks or now putting hands to work to get an incremental backup. But I'm sure some people would enjoy a free, open source solution to make those incremental backups out of the box. I understand that this isn't the goal of rsync. I'm only pointing this because the adaptation would only take few code lines and I think rsync worth this, as an option --incremental or something like that that people are not obligated to use by default. If you still think rsync don't need it, the solution would be giving another name for the moded rsync. Unfortunately I can't help going this way. I could have some time to code this feature inside rsync, but maintaining a parallel program takes too much resources for me. Best regards, Vitorio-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another rsync incremental thread
On Sat, 2008-07-19 at 15:45 +0200, Mac User FR wrote: > For rsnapshot, OK, I understood now how to connect them. The "post- > xfer exec" I was looking for don't exist on the rsync man page, but > on the rsyncd.conf which I've didn't look at (my fault :/ ). I think I > will not use rsnapshot because I don't like to use to many software > but only a small rotating script home made. > > Didn't finished the script yet, but it will do something like this: > 1) Expected first state: > > backup (rsync transfer folder), lastest (previous transfer folder) and > directories (old backup) > > 2) Execution of the script (pseudo code): > > get date > get free disk space > get backup size > > mv lastest -> > mv backup -> lastest > > while(free space < backup size) #disk almost full, need deleting some > old stuff > get oldest backup folder name > rm it > > That's all. The only thing I haven't coded yet is the while loop, but > I will do it next week. > What could rsnapshot give me in addition to this? Would it manage free > space for many concurrent backups? > Using my script for 2 or more backups, if the other backup takes > almost all disk, it won't be smart enough to delete backups from the > other backup. There are some other cases I should care about. But I > still have the impression rsnapshot would be too big stuff for only > this need. Rsnapshot is similar to your script except that it retains a specified fixed number of backups. It has no feature to delete as many backups as necessary to free the proper amount of disk space, but that could be done pretty easily in a script that is invoked before (or after) rsnapshot. With multiple clients, it would not be hard to make the script consider all the snapshot roots together when choosing the backup to delete. I think rsnapshot is not as heavy-weight as you believe, but go ahead and use your own script if you find it easier to understand. > In another hand, don't you think we could add the content of this > script (revised, of course) to make rsync a complete incremental > backup solution without needing 3rd part software? Isn't a good idea? Why? The goal of rsync is to copy files, not to be a complete incremental backup solution by itself, even though several of its options are motivated by this use case. I think the current state of affairs with a wide variety of available rsync-based backup solutions (rsnapshot, dirvish, ccollect, various homegrown scripts including yours, ...) is just fine. Different users prefer different solutions, so I don't think arbitrarily choosing one to include in rsync would be appropriate. Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another rsync incremental thread
Thanks Matt!! I'm always amazed of the quick and precise answers you give! Yes, sorry, I meant push (English isn't my native langage and I always swap push and pull). For rsnapshot, OK, I understood now how to connect them. The "post- xfer exec" I was looking for don't exist on the rsync man page, but on the rsyncd.conf which I've didn't look at (my fault :/ ). I think I will not use rsnapshot because I don't like to use to many software but only a small rotating script home made. Didn't finished the script yet, but it will do something like this: 1) Expected first state: backup (rsync transfer folder), lastest (previous transfer folder) and directories (old backup) 2) Execution of the script (pseudo code): get date get free disk space get backup size mv lastest -> mv backup -> lastest while(free space < backup size) #disk almost full, need deleting some old stuff get oldest backup folder name rm it That's all. The only thing I haven't coded yet is the while loop, but I will do it next week. What could rsnapshot give me in addition to this? Would it manage free space for many concurrent backups? Using my script for 2 or more backups, if the other backup takes almost all disk, it won't be smart enough to delete backups from the other backup. There are some other cases I should care about. But I still have the impression rsnapshot would be too big stuff for only this need. In another hand, don't you think we could add the content of this script (revised, of course) to make rsync a complete incremental backup solution without needing 3rd part software? Isn't a good idea? Best regards, Vitorio Le 18 juil. 08 à 21:35, Matt McCutchen a écrit : On Fri, 2008-07-18 at 12:40 +0200, macuserfr wrote: What's new? On my new job I have several servers to administrate. Servers that aren't backed up (sic). So, there's why I'm back to rsync. The backup plan I would like: 1) Client side: PCs running rsync (or cwrsync with UTF-8 mod for long names for windows ones). They run rsync (I don't need data encryption so no need to ssh unless it's simpler) and pull data they want to the backup server. This side is OK. I think you mean "push". 2) Server side: A standard PC with FreeNAS FreeBSD distrib, RAID-1 disks and rsync server. Here I would like the backups to be incremental and rotated, with a sort of time stamp. Here is where things get complicated. Just use an rsync daemon wired up to an rsnapshot installation in sync_first mode, where the daemon module points to the .sync dir in the snapshot root and the "post-xfer exec" command calls "rsnapshot LOWEST-INTERVAL". You can use a listening daemon or a single-use daemon over ssh as you prefer. This is the approach I've been recommending for pushing hard-linked backups for a while; it achieves everything you want without modifying any of the tools. I helped Eric Johansson with a similar setup a while ago: http://lists.samba.org/archive/rsync/2007-December/019470.html If you have multiple clients pushing backups, it's simplest to have a separate module and snapshot root for each client. To keep things manageable, you can write a script to automatically generate the individual rsnapshot.conf files by substituting the necessary values into a template. Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another rsync incremental thread
On Fri, 2008-07-18 at 12:40 +0200, macuserfr wrote: > What's new? On my new job I have several servers to > administrate. Servers that aren't backed up (sic). So, there's why I'm > back to rsync. > > The backup plan I would like: > > 1) Client side: PCs running rsync (or cwrsync with UTF-8 mod for long > names for windows ones). They run rsync (I don't need data encryption > so no need to ssh unless it's simpler) and pull data they want to the > backup server. This side is OK. I think you mean "push". > 2) Server side: A standard PC with FreeNAS FreeBSD distrib, RAID-1 > disks and rsync server. Here I would like the backups to be > incremental and rotated, with a sort of time stamp. Here is where > things get complicated. Just use an rsync daemon wired up to an rsnapshot installation in sync_first mode, where the daemon module points to the .sync dir in the snapshot root and the "post-xfer exec" command calls "rsnapshot LOWEST-INTERVAL". You can use a listening daemon or a single-use daemon over ssh as you prefer. This is the approach I've been recommending for pushing hard-linked backups for a while; it achieves everything you want without modifying any of the tools. I helped Eric Johansson with a similar setup a while ago: http://lists.samba.org/archive/rsync/2007-December/019470.html If you have multiple clients pushing backups, it's simplest to have a separate module and snapshot root for each client. To keep things manageable, you can write a script to automatically generate the individual rsnapshot.conf files by substituting the necessary values into a template. Matt signature.asc Description: This is a digitally signed message part -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html