Re: Backup scripts - recycling old backup directories
I'm using rsync to backup file systems from one machine to another (not sure why we aren't using amanda like we are everywhere else). The combination of rsync updating blocks, rather than replacing files or replicating file systems is a huge savings for us. As daily zfs snapshots (saved for a day, a week, a month, three months) on the receiving end provides a nice recovery window. On Fri, Sep 12, 2014 at 02:31:47PM +1000, Robert Bell wrote: Folks, Kevin Korb wrote: Have you considered more advanced methods such as subvolume snapshots provided by ZFS and BTRFS? At work we were forced to abandon rsync - --link-dest because of the amount of time it takes to delete old backups when the data is primarily many millions of small files (shared web hosting company). We don't have more advanced methods like subvolume snapshots available to us. However, we can recycle backup directories. When we started using rsync with --link-dest back in about 2007, we deleted old backups, but realised soon after that we could recycle old backups. With daily backups, we find about 1.5% of the data and 0.5% of the files change from one day to the next, so a directory from about 5 days ago will typically be only 5-10% wrong and can be recycled to be the target of the latest directory - that's a lot better than recreating the whole directory tree for a new backup, and then deleting a whole old directory tree. We use --delete of course. Hope this helps someone. Rob. Dr Robert C. Bell HPC National Partnerships | Scientific Computing Information Management and Technology CSIRO T +61 3 9669 8102 Alt +61 3 8601 3810 Mob +61 428 108 333 robert.b...@csiro.aumailto:robert.b...@csiro.au | www.csiro.au | wiki.csiro.au/display/ASC/ Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html --- Brian R Cuttler brian.cutt...@wadsworth.org Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync between OpenVMS OpenVMS
On Thu, Aug 16, 2012 at 09:24:56PM +, Horn, James wrote: I'm looking at need to duplicate some drives data from one OpenVMS site to another OpenVMS site. Clustering is not an option, so was wondering if Rsync could be used between two OpenVMS sites and if there would be any issues with the files. I've never installed rsync on VMS, but don't forget that you can use $ backup. It will work fine and will reproduce the tree structure, but will only allow a single thread across the network. That is, $ backup on the local system and output a save_set on the remote end, you can then unpack the save_set on the remote system. James Horn SHSU 2449 Computer Services Sam Houston State University Huntsville TX 77340 Phone: (936) 294-1042 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html --- Brian R Cuttler brian.cutt...@wadsworth.org Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773 IMPORTANT NOTICE: This e-mail and any attachments may contain confidential or sensitive information which is, or may be, legally privileged or otherwise protected by law from further disclosure. It is intended only for the addressee. If you received this in error or from someone who was not authorized to send it to you, please do not distribute, copy or use it or any attachments. Please notify the sender immediately by reply e-mail and delete this from your system. Thank you for your cooperation. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: feature request: remote user is root, make remote owner is foo
On Wed, Aug 11, 2010 at 02:51:35PM -0700, travis+ml-rs...@subspacefield.org
wrote:
On Wed, Aug 11, 2010 at 01:32:42PM -0400, Brian Cuttler wrote:
[Set u+s on directories, don't worry about owners]
It seems to work relatively well. I get an error about not being
able to chgrp the files owned by other users, and, in my case,
the group ends up wrong because it's not supposed to be the same
on both ends (www-data on destination, something else on source).
However, because of the former problem, at least the extant files
stay readable until I can fix-up the group problem caused by the
latter. If I further chown them to www-data, then they don't get
their gid hosed the second time around.
Not an ideal solution, but seems workable for the time being.
Aside: chown really needs a flag that says set GID on directories but
nothing else. Right now I do this, which is the only right thing to
do when filenames might contain anything, including whitespace or
newlines:
We use a # find command to find directories and feed it to
the chown action. I'd have to think about/look-up the exact
syntax, but its a one line command and runs fairly quickly
since you can get the find to do the file selection (find directories)
rather than passing that work further down the pipe.
setperms () {
local perms=$1
local fileperm=$2
local dirperm=$3
local own=$4
local dir
shift 4
for dir in $@
do
if test -d $dir
then
find $dir -xdev -type d -print0 | xargs -0 --no-run-if-empty chmod
$perms,$dirperm --
find $dir -xdev -type f -print0 | xargs -0 --no-run-if-empty chmod
$perms,$fileperm --
chown -R $own $dir
fi
done
}
setperms u=rwX,o=rX g=rwX g=rwxs owner:group /path/to/whatever
--
A Weapon of Mass Construction
My emails do not have attachments; it's a digital signature that your mail
program doesn't understand. | http://www.subspacefield.org/~travis/
If you are a spammer, please email j...@subspacefield.org to get blacklisted.
---
Brian R Cuttler brian.cutt...@wadsworth.org
Computer Systems Support(v) 518 486-1697
Wadsworth Center(f) 518 473-6384
NYS Department of HealthHelp Desk 518 473-0773
IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: feature request: remote user is root, make remote owner is foo
Travis, As a matter of principle, SOP, we don't like to ssh/rsync as root and generally don't allow root ssh/rsync into a box. Better/safer to move the security stuff to a lower powered user if you can. On Wed, Aug 11, 2010 at 10:18:11AM -0700, travis+ml-rs...@subspacefield.org wrote: I often push files from my user account over SSH to my web server, and want them owned by www-user, which may not have a login shell, should never accept remote logins, and who may not have a ~/.ssh directory (and if it did, it would be under the wwwroot, ack!). Currently I push as root and then do a chmod, but isn't there a better way? While I'm doing this, the files are temporarily unavailable, since they aren't readable by www-user as they exist on the local system. -- A Weapon of Mass Construction My emails do not have attachments; it's a digital signature that your mail program doesn't understand. | http://www.subspacefield.org/~travis/ If you are a spammer, please email j...@subspacefield.org to get blacklisted. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html --- Brian R Cuttler brian.cutt...@wadsworth.org Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773 IMPORTANT NOTICE: This e-mail and any attachments may contain confidential or sensitive information which is, or may be, legally privileged or otherwise protected by law from further disclosure. It is intended only for the addressee. If you received this in error or from someone who was not authorized to send it to you, please do not distribute, copy or use it or any attachments. Please notify the sender immediately by reply e-mail and delete this from your system. Thank you for your cooperation. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: feature request: remote user is root, make remote owner is foo
Travis, We also use rsync to push our files. While there are several users with the ability to do the push, the files on the webserver host are set with su-gid bit set. No matter which of our web people push the files to the visible server the files all move to a consistent groupship that allows the other users to replace them later. You may do something similar with your setup, the webserver group being in the same group as your webmaster(s)', at least on the receiving end of things. Are files on webserver are 775 and directories 1775 and we are less concerned about the owner of the file than the group of the file. YMMV Brian On Wed, Aug 11, 2010 at 10:18:11AM -0700, travis+ml-rs...@subspacefield.org wrote: I often push files from my user account over SSH to my web server, and want them owned by www-user, which may not have a login shell, should never accept remote logins, and who may not have a ~/.ssh directory (and if it did, it would be under the wwwroot, ack!). Currently I push as root and then do a chmod, but isn't there a better way? While I'm doing this, the files are temporarily unavailable, since they aren't readable by www-user as they exist on the local system. -- A Weapon of Mass Construction My emails do not have attachments; it's a digital signature that your mail program doesn't understand. | http://www.subspacefield.org/~travis/ If you are a spammer, please email j...@subspacefield.org to get blacklisted. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html --- Brian R Cuttler brian.cutt...@wadsworth.org Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773 IMPORTANT NOTICE: This e-mail and any attachments may contain confidential or sensitive information which is, or may be, legally privileged or otherwise protected by law from further disclosure. It is intended only for the addressee. If you received this in error or from someone who was not authorized to send it to you, please do not distribute, copy or use it or any attachments. Please notify the sender immediately by reply e-mail and delete this from your system. Thank you for your cooperation. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Anyone syncing database files?
I second that. We backup Oracle on Solaris but we insist that the oracle is down. On Fri, Aug 20, 2004 at 10:48:20PM +0200, Jan-Benedict Glaw wrote: On Fri, 2004-08-20 16:43:07 -0400, Loukinas, Jeremy [EMAIL PROTECTED] wrote in message [EMAIL PROTECTED]: Anyone syncing say 18-20gb Oracle dbf files..? No problem--as long as you don't try to backup hot database files. Shutdown Oracle first:) MfG, JBG -- Jan-Benedict Glaw [EMAIL PROTECTED]. +49-172-7608481 _ O _ Eine Freie Meinung in einem Freien Kopf| Gegen Zensur | Gegen Krieg _ _ O fuer einen Freien Staat voll Freier Bürger | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) ~(NEW_COPYRIGHT_LAW | DRM | TCPA)); -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html --- Brian R Cuttler [EMAIL PROTECTED] Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync'ing large files
I can't address the algoritm questions but I'll tell you that we had a tremendous improvement is speed when we switched to a newer version of rsync. We are using it (in this case) to rsync our oracle files to a separate partition on the system cpu. I'm using rsync to copy some large (1GB) oracle datafiles. I've noticed that sometimes it transfers some of the files twice. Some earlier posts to this list that I saw in the archives seemed to indicate that this is a problem with the rsync algorithm itself when dealing with large files. Some of the mails seemed to indicate that this can be mitigated by using larger block sizes, though there were some caveats that increasing block size without increasing checksum size might cause more hash collisions. My questions: 1) Can anyone explain the problem to me in layman's terms. Is the initial bad transfer due to hash collisions? 2) If I'm transferring files that are 1-2GB, would increasing the block-size parameter to 8k or so help here? Or would I be creating more chances for hash collisions since I can't increase the checksum size? 3) I'm using 2.5.5 (yeah, ancient I know, I'll be upgrading it soon). Are later versions better at dealing with this problem? Any help is appreciated! Thanks, Jeff -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync is great
Use # find on the remote system and list | rm files that meet your date criteria ? I'm not on the list but I just wanted to throw my positive feedback out there. I've been using rsync to backup work that I generate within a lab to an offsite system. If I lost this data ... So great. In standard community fashion, though, I want to throw an idea out there. It would be nice if there were an option like: --delete-if-too-old Currently I run a cron job which copies files offsite nightly. Once a month, though, I use the --delete option to get rid of cruft. It would be _nice_, though, if I could delete things that were older than a month instead of a simple comparison between what's on the sender and the receiver. This way files would have a known age of 30 days or whatever before they get tossed. As it is, the interval I can regress to has to do with the time of the month, which is just too close to real life ... If there is a better way to submit suggestions I will do so upon hearing about it. Thanks for rsync. I use it every day. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Logging from cron
We always run rsync from a shell script, sometimes we pipe the shell script output to mailx or such. I'm sure you can write it to a log file but don't recall the syntax off hand. We never run the single command as a cron task, ie never 0 23 * * 1-5 rsync but do 0 23 * * 1-5 script1.sh I'm sure a number of the OS provided root cron jobs do so. [Charset KOI8-R unsupported, filtering to ASCII...] Hello, I've just spent several hours going over several Google searches trying to find a way to configure rsync to log into a file named /var/log/rsync.log. So far, every instance where I've found someone asking about rsync logging remained unanswered (which is kind of weird in itself). As far as I can tell, the only way to do this is to setup rsync as a daemon process so that it will read an rsync.confg. The problem is that I don't want to run rsync as a daemon process, and am not interested in working from a rsync.confg file if I can avoid it. My rsync command lines are being generated on the fly by a perl program running via cron, and then shelling the command to the OS (linux) and handshaking via SSH. This is working fine, but I need a log to determine what is causing rsync to shutdown before completing the full mirror of the server. Does anyone have any suggestions as to how I can get a log generated from rsync running from the command line? Many Thanks in Advance, Tim -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync hang
Hello rsync list, I am running a Solaris 8 system and have rsync 1.6.2 protocal 14 installed. We have updated (wed 17-dec) to the latest Solaris 8 recommended patch set. When performing an rsync of two local UFS partitions rsync hung, never completed, didn't exit. What I forced an exit via kill signal core dumped. Config of disks is that 1) Mirror by Solistice Disksuite rsync to non-mirrored UFS partition. 2) This non-mirrored UFS partition is later copied to a second non-mirrored partition. It was during this second stage that rsync hung. There was nothing particularly odd about the files being moved, not size nor content. They where in fact the unzip of the 8_Recommended patch set. We have been coping one partition to another in this mannor for several months using this procedure (extracted from larger procedure). echo Copy /bk1-dbX files to /bk2-dbX foreach DBX (db1 db2 db3) set FDIR=/bk1-$DBX/oracle set TDIR=/bk2-$DBX/oracle echo Copy of $DBX files from bk1 to bk2 echo $FDIR $TDIR #rsync -rltv --delete -e ssh $FDIR $TDIR rsync -rltv --delete $FDIR $TDIR echo echo end This problem has only occured once but because the intent is to copy our database files (Oracle) and the script also performs a database shutdown and restart its something we felt we'd best meantion to the list. The database didn't restart because the script didn't complete the rsync and processing is sequential. So its not just a matter of the rsync error, production was down. Am I abusing some flag or is there a limit on number of files, size, something that I inadvertantly hit ? I'm thinking its an anomaly, rsync successfully moved the same data from disk a-b, has been running very well for months. We did find a memory error in /var/adm/messages - but the time stamp was off by many hours, so I'm thinking that is unrelated. There are no reported disk error, disksuite reports that it is in good shape as well. All disks are local. Thanks for your assistance, if more info is needed please ask. Brian --- Brian R Cuttler [EMAIL PROTECTED] Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync hang
J.W. The memory error was a correctable logged at 15:30 PM, with the apparent time on the rsync issue being 23:00. It would be nice to see everything tie together but I don't see it this time (I've have pushed the same conclusion if I where you. And, yes, I understand the humor - subtext is sooo difficult in email). Thank you for the compliment on the bug-report. We will look to install the newer version of rsync, its just the right thing to do. 2.5.7 has been around a while and is stable or do you think we should wait for 2.6.0 ? Any interest in the core file ? thanks and happy holidays, Brian --- Brian R Cuttler [EMAIL PROTECTED] Computer Systems Support(v) 518 486-1697 Wadsworth Center(f) 518 473-6384 NYS Department of HealthHelp Desk 518 473-0773 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
