Re: Backup scripts
Hi, I think the words used are not correct, that's why you (and Kevin) are a bit confused. You are in fact asking for incremental backup, not a snapshot : http://en.wikipedia.org/wiki/Snapshot_(computer_storage). In this case, you can use rsync with a few flags to delete (or move in another directory) the modified/deleted files regarding the last backup. This would look something like this ((h)uman, do not (x)cross filesystem boundaries, (P)rogress, (H)ard-links, (a)rchive, aka recursive+symlinks+perms+time+owner+group+specials and g(z)ip): # Cloning your data to a distant folder with timestamp rsync -hxPHaz --exclude=exclude_list source target/$NOW/ # Incremental backup only rsync -hxPHaz --exclude=exclude_list --delete --backup --backup-dir=backup source target/latest/ # You can even suffix the modified/deleted files rsync -hxPHaz --exclude=exclude_list --delete --backup --backup-dir=backup --suffix=~$NOW source target/latest/ Hope that will help, Greg On Thu, Sep 4, 2014 at 11:21 PM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yes, you have different scripts now at least. But I still don't see any snapshotting. The only difference in these is that the full one rsyncs to a date+time stamped directory while the snapshot one rsyncs to a directory named snapshot. As far as I can tell both will be a complete copy with no relationship to the other. On 09/04/2014 05:17 PM, Chris wrote: On Thu, 2014-09-04 at 17:00 -0400, Kevin Korb wrote: Do you actually have any snapshots currently? From the scripts you posted it seems to just be rsyncing to the same dir every run and only claiming to be making snapshots. If you do have snapshots now then something else is happening in addition to this script. It was pointed out to me in a direct message that I had inadvertently posted the same scripts (full backup) in pastebin. Here are the correct one. Full http://pastebin.com/dEk7kBip Snapshot http://pastebin.com/H7SuABN1 On 09/04/2014 04:58 PM, Chris wrote: On Thu, 2014-09-04 at 12:46 -0400, Kevin Korb wrote: The scripts you posted look the same to me. And I don't see any form of snapshotting. The $NOW variable is set and is echoed but it is never actually used. For an rsync snapshot I would expect to see either rsync --link-dest or a cp -al depending on the age of the script. The only reason to need two scripts would be because the snapshot would need 3 dirs to work with (the source, the target, and the previous backup) but that could also be handled with 1 script and a simple existing check. On 09/04/2014 12:30 PM, Chris wrote: I have two scripts that a kind soul on this list wrote for me over 4yrs ago. I got to looking at them the other day because my old box crashed and had to build a new one also got a new backup USB drive since I'm still copying over things from the old one. The first one is for a full backup: http://pastebin.com/XF6Zm42A Works great, does exactly what it's supposed to do. The second is for a 'snapshot' which is where I get a bit confused. I would think that a 'snapshot' would be just the changed files either since the last full backup or since the last 'snapshot' the night before. It seems though that it's actually the same as a full backup. I don't profess to be a script person so I have no idea if it's doing what it should or something needs to be changed. Below is the 2nd script: http://pastebin.com/MkBzJnux Any advice would be appreciated. Chris Thanks Kevin, I guess for now I'll leave them as they are until I can get smart on scripting. Chris - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlQI19QACgkQVKC1jlbQAQcn9QCfdyjYePz5IanuxcowLuRcBnIN tpcAoNRVEndI3F6+we8rSHASybAWd471 =W5WZ -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Prevent dereferenced path of symlinked directories from printing on client side log output
Hello, I found something that could be interesting for you : the *command=* that you can put at the beginning of the corresponding line of your $HOME/.ssh/authorized_keys. For this to work, you must use rsync with rsa/dsa keys. $HOME/.ssh/authorized_keys should have one line like this on the server host : command=rsync --server --sender -Phaz --copy-links --copy-dirlinks . /backup/path/ ssh-rsa YourVeryLongSSHKey... Note that this will override all options passed to rsync on the client side. See this link : http://learninginlinux.wordpress.com/2009/05/07/rsync-fixed-server-side-options/ Greg On Tue, Dec 18, 2012 at 10:13 AM, wesley wes...@tormail.org wrote: I'm trying to figure out if there is some way to run an rsync daemon which makes a mix of real and symlinked folders available to a client such that all the folders appear as real to the client. I realize the --copy-links and --copy-dirlinks accomplish this when invoked from the client side (and that is currently how I am accomplishing this). Is there any way to essentially force those options from the daemon end? So that to a client, a symlinked dir will always appear as the referent dir. If not, then if at all possible, I would like to block the referent dir from showing up in the log output on the client side. If the client is printing --progress output to the terminal, or is tailing the log output, rsync will indicate when a directory is a symlink and print the dereferenced directory. Is there any way to prevent rsync from printing drreferenced directory info? Printing this info to the client can leak potentially sensitive information about the filesystem/usernames/etc on the server running rsyncd, which is part of the reason why symlinks are being used (to present a consistent clean set of read only dirs that may be downloaded from). Presenting the referent dir paths leaks information that it would be preferable not to leak. Thanks to anyone that can point me in the right direction (or letting me know if it's not possible). If it's not possible, would very much appreciate if it could be considered as a feature enhancement to the rsyncd because of the privacy implications of leaking referent path data to clients. -- Wesley -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/** mailman/listinfo/rsync https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/**smart-questions.htmlhttp://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync using huge traffic
Hello, First things I'm thinking of is : 1. Do you have inside your backup tree some links that point outside the filesystem ? (ie. some mounted folder) If so, add the -x flag (don't cross filesystem boundaries) 2. Do you have some hard links inside your tree that may create recursive loops ? If so, try with the -H flag too (preserve hard links) That would lead to -xHaze 'ssh ...' Greg On Thu, Nov 29, 2012 at 9:24 AM, Ben Oswald benosw...@benoswald.de wrote: Hey, I'm using rsync to backup my server but there is a problem because rsync is using very huge amounts traffic. But first to the setup. The server I backup has 4GB of data and I use the following command to backup this data. /usr/bin/rsync -aze 'ssh -i /root/.ssh/backup.key -l backupuser' --rsync-path='sudo rsync' --delete --exclude-from=ex.list $SRC $TRG The problem is that the traffic generated to backup these 4GB of data is often (yes not every time I backup) up to 200GB and more if I don't stop the process. Rsync it self is showing me not more than 4GB of transmitted data at the end of the rsync call. So is there a known reason for this behaviour of rsync and if yes how can I solve this problem? Best regards, Ben -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: collecting the differences between a local host and remote archive to third local location (a usb device)
Hi, So in some ways, you would like to use rsync as a diff/patch tool. There are some similar questions out there (herehttp://serverfault.com/questions/62364/get-rsync-to-generate-a-patch-file-instead-of-copying-across-files or there http://lists.samba.org/archive/rsync/2005-January/011439.html), and the answers are quite similar too. Basically, two methods : I. Bash scripting (1. Do a dry run and export the files list, 2. Get only these files on your USB, 3. Push the new/updated files on the original system) II. Batch-mode (see this examplehttp://www.linuxmantra.com/2011/02/rsync-in-batch-mode.html ) Hope that will help, Greg On Thu, Sep 27, 2012 at 9:33 AM, Benjamin Ward b...@forward.net.au wrote: Hi everyone, I'm trying to archive to a remote host but the size of the data to copy is prohibitively large to consider doing the sync over the wire (it's TBs of raw video), so i was wondering if it was possible to use a usb attached storage device as the transport medium. Is it possible for rsync to compare the two systems (over the network) finding any newer or different files on the local system as compared to the remote archive (ignoring local deletes), then copy only those new/differences to a locally attached portable drive so that the portable drive can be physically taken across town to the other system, plugged in and then the new data be ingested so that a subsequent rsync between the the two systems would see those files as synchronised? I thought the --compare-dest option might be the way to go, as per this post: http://lists.samba.org/archive/rsync/2006-June/015827.html but it seems to not behave the way i thought it would. Thanks in advance, and apologies if I'm asking a bleeding obvious question. :) Kind regards, Ben -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: cannot rsync when source directory lacks write permission
Hi, No I think this bug is not related. However, I am wondering : if the directory is write-protected, even for root, that means it never changes right ? or is it that there is no file creation, only updates ? In either case, the better would be to treat this particular directory separately. Suppose this directory is called secure, you would create a secure folder on the destination, with a chmod +w, then do : rsync $SRC $DST -x secure\* rsync $SRC/secure $DST/secure/ Greg On Tue, Aug 7, 2012 at 7:57 AM, András Porjesz andras.porj...@ericsson.comwrote: Hi, Probably it is related to this bug: https://bugzilla.samba.org/show_bug.cgi?id=8844 Andras -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync-boun...@lists.samba.org] On Behalf Of Steven Levine Sent: Friday, July 27, 2012 20:15 To: rsync@lists.samba.org Subject: Re: cannot rsync when source directory lacks write permission In juuivm$ai5$1...@dough.gmane.org, on 07/27/12 at 01:26 PM, Brian J. Murrell br...@interlinx.bc.ca said: Hi, I seem to be running into a problem where I am trying to rsync from a source directory that lacks write permissions (i.e. r-xr-xr-x). Presumably this is because rsync creates the directory on the destination, then sets the permissions to match the source and then tries to sync the contents of the directory, which it cannot of course lacking write permission in the directory. Rsync contains code to handle this. What version of rsync are you running and what is the command line you are using and what is the exact error message you are getting? Regards, Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync like Time Machine
Hi, As for the destination directory and the backup directory (--backup-dir), rsync will create the missing subdirectory (one level below the existing dir only), so yes for /somedir-exists/newdir-with-date, no for /somedir-exists/newdir-with-year/newdir-with-month on january 1st... But if you want this dir to be a symlink, you can't. Greg On Fri, Jul 27, 2012 at 7:16 PM, Clint Olsen clint.ol...@gmail.com wrote: I've been very interested in these discussions and uses of rsync as a clone of Time Machine. A couple of things have been keeping me from a fully automated solution. I'd like to eliminate the need for Samba/NFS mounts of any kind, because they have proven to be unreliable for me and under some operating environments (Cygwin) it breaks --link-dest. In most of the articles I've read, a target date directory is created with some sort of latest symlink for the --link-dest parameter. I can accomplish those tasks via remote ssh commands, but I was hoping there was a better way. For example, is there any circumstance where you can coax rsync into creating a target directory that's not there already? % rsync source user@nas::module/somedir-exists/newdir-with-date So, newdir-with-date doesn't exist (yet). I would like to have rsync create it for me. Is that even possible? Thanks, -Clint -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync like Time Machine
If you really want to have a destination tree that looks like : current - 2012-07-22 2012-07-22/ 2012-07-21/ with the current symlink pointing to the latest backup, you can manage to do it in two passes : 1. Create an empty directory '2012-07-22/' and the 'current' symlink pointing to it (relative path), and rsync it to the final destination in replace mode (ie. update, no deletion) with symlink on (-l). This will replace the existing 'current' symlink in the destination folder. 2. Rsync (-Ha or -Haz) your src folder to 'current/' or to '2012-07-22/' (leave the trailing slash) Greg On Fri, Jul 27, 2012 at 10:06 PM, Clint Olsen clint.ol...@gmail.com wrote: Ok, that is helpful. As you can guess based on my question, it would be nice if all the automation can be done on the client side rather than having some specialized scripting on the receiving side to manage directories and symlinks etc. Thanks, -Clint On Fri, Jul 27, 2012 at 12:55 PM, Greg Deback (rsync) greg.deb+rs...@gmail.com wrote: Hi, As for the destination directory and the backup directory (--backup-dir), rsync will create the missing subdirectory (one level below the existing dir only), so yes for /somedir-exists/newdir-with-date, no for /somedir-exists/newdir-with-year/newdir-with-month on january 1st... But if you want this dir to be a symlink, you can't. Greg On Fri, Jul 27, 2012 at 7:16 PM, Clint Olsen clint.ol...@gmail.comwrote: I've been very interested in these discussions and uses of rsync as a clone of Time Machine. A couple of things have been keeping me from a fully automated solution. I'd like to eliminate the need for Samba/NFS mounts of any kind, because they have proven to be unreliable for me and under some operating environments (Cygwin) it breaks --link-dest. In most of the articles I've read, a target date directory is created with some sort of latest symlink for the --link-dest parameter. I can accomplish those tasks via remote ssh commands, but I was hoping there was a better way. For example, is there any circumstance where you can coax rsync into creating a target directory that's not there already? % rsync source user@nas::module/somedir-exists/newdir-with-date So, newdir-with-date doesn't exist (yet). I would like to have rsync create it for me. Is that even possible? Thanks, -Clint -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync pairing
Hello, You could simply create a dedicated group, set up permissions to your script, and add only the desired remote users to that group. Greg On Tue, Jun 26, 2012 at 2:18 PM, András Porjesz andras.porj...@ericsson.com wrote: Hi all, I have an rsyncd installed on a host and I want to use it from a lot of clients. This rsync is not invoked directly but there is a shell script wrapper co set up parameters and environment. My problem is that I do not want to allow any other rsync to be able to connect to the daemon. Is there any way to implement such feature? Thanks Andras -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync pairing
And you can also restrict users given their IPs : Add at the top of the rsyncd.conf hosts allow = 10.42.20.43/32 Greg On Tue, Jun 26, 2012 at 2:38 PM, Greg Deback (rsync) greg.deb+rs...@gmail.com wrote: Hello, You could simply create a dedicated group, set up permissions to your script, and add only the desired remote users to that group. Greg On Tue, Jun 26, 2012 at 2:18 PM, András Porjesz andras.porj...@ericsson.com wrote: Hi all, I have an rsyncd installed on a host and I want to use it from a lot of clients. This rsync is not invoked directly but there is a shell script wrapper co set up parameters and environment. My problem is that I do not want to allow any other rsync to be able to connect to the daemon. Is there any way to implement such feature? Thanks Andras -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync pairing
Okay, then you can do the following : - create a dedicated group rsync - chown/chmod /bin/rsync for that group only - add a setgid to your script with that group - use IP control in the rsyncd.conf to restrict users Thus only allowed users can access the script, and only the script can launch rsync Greg On Tue, Jun 26, 2012 at 2:41 PM, András Porjesz andras.porj...@ericsson.com wrote: ** Hi, thanks, but actually the same users want to use rsync without the wrapper and it will cause inproper transfer. Andras -- *From:* greg@gmail.com [mailto:greg@gmail.com] *On Behalf Of *Greg Deback (rsync) *Sent:* Tuesday, June 26, 2012 14:38 *To:* András Porjesz *Cc:* rsync@lists.samba.org *Subject:* Re: Rsync pairing Hello, You could simply create a dedicated group, set up permissions to your script, and add only the desired remote users to that group. Greg On Tue, Jun 26, 2012 at 2:18 PM, András Porjesz andras.porj...@ericsson.com wrote: Hi all, I have an rsyncd installed on a host and I want to use it from a lot of clients. This rsync is not invoked directly but there is a shell script wrapper co set up parameters and environment. My problem is that I do not want to allow any other rsync to be able to connect to the daemon. Is there any way to implement such feature? Thanks Andras -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync files from subfolders on source to root of a folder on destination
If I were you, I would start by creating a unique folder, the image of
Destination/, and fill it with symlinks to your flac files, using -exec,
*then* calling rsync only once with the -L option (--copy-links). Thus you
should be able to benefit from the --delete option and keep an up-to-date
destination tree... but take care of the corrupted symlinks (another -exec
loop?).
Greg
On Mon, Apr 23, 2012 at 8:50 AM, James Robertson j...@mesrobertson.com wrote:
I wish to sync a bunch of flac files that reside in various subfolders
to the root of a folder on a destination.
An example of the directory structure on the source is:
source tree Music/
Music/
├── R
│ ├── Radiohead
│ │ └── OK Computer
│ │ ├── 01 - Radiohead - Airbag.flac
│ │ ├── 02 - Radiohead - Paranoid Android.flac
│ └── Red Hot Chilli Peppers
│ └── Greatest Hits
│ ├── 01 - Red Hot Chili Peppers - Under the Bridge.flac
│ ├── 02 - Red Hot Chili Peppers - Give It Away.flac
I am using this command which syncs fine but includes the directory
paths on the destination:
rsync -rltDzvh --delete -e ssh Music/ user@192.168.1.1:/Destination/
--include=*/ --include=*.flac --exclude=*
So on the destination the structure is:
destination tree /Destination/
/Destination/
├── R
│ ├── Radiohead
│ │ └── OK Computer
│ │ ├── 01 - Radiohead - Airbag.flac
│ │ ├── 02 - Radiohead - Paranoid Android.flac
│ └── Red Hot Chilli Peppers
│ └── Greatest Hits
│ ├── 01 - Red Hot Chili Peppers - Under the Bridge.flac
│ ├── 02 - Red Hot Chili Peppers - Give It Away.flac
I want to prune all directories so only the files are placed into the
root of /Destination/ e.g.
destination tree /Destination/
/Destination/
├── 01 - Radiohead - Airbag.flac
├── 02 - Radiohead - Paranoid Android.flac
├── 01 - Red Hot Chili Peppers - Under the Bridge.flac
├── 02 - Red Hot Chili Peppers - Give It Away.flac
All the files have different names so that's ok and I have reviewed
the various options in rsync such as --no-relative but have been
unable to get it working as desired.
I also came up with:
find ./Music/ -name *.flac -exec rsync -ltDzvh {} -e ssh
user@192.168.1.1:/Destination/ \;
But this means using ssh keys and is probably inefficient and if I
want to use the --delete it would likely break everything.
So how would I achieve this whilst still being able to use --delete or
--delete-excluded for times when I add, remove or rename items on my
Music library on the source and have those changes sync to the
destination?
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: backup to NTFS USB disk
Hello all, I don't know if this would really help, but I had some problems with latin characters too while syncing between a Windows NT fileserver (NTFS) and a linux box (ext3), and I have solved it using --iconv option, as follow : rsync -hxPHaz -e 'ssh' --stats --iconv=CP1252 --delete --backup --backup-dir=$BKP_DIR $SRC $DEST (-h: human readable, -x: don't cross filesystems, -P: --partial --progress, -H: hard links, -a: archive, ie. own/group/mod/time/..., -z: compression) Greg On Sun, Apr 15, 2012 at 10:05 AM, joop g j...@xs4all.nl wrote: ** On Friday 13 April 2012 21:22:42 jose...@main.nc.us wrote: Since no one has replied yet, I have one idea that *might* point to part of your problem. I've never had to deal with locale issues, so I have no idea about that. You have directories and file names with blanks in them. In general, this causes a lot of trouble for a lot of programs. I'm not sure how rsync handles them. It is very kind of you to at least give it a try, but I am quite sure that spaces are not the problem. In fact, I have been experimenting further, and the situation is even more complicated. First of all, I have to apologize to the list for my first question, regarding WARNING: Couldn't set locale to 'nl_NL.iso-9959-1' thus some file names may not be correct or visible. Please see the potential solution at http://ntfs-3g.org/support.html#locale I could try, of course, but it was clearly off-list. Before I am to present my more complex problem, I might give some explanation. I am managing the computer systems for a small regional museum, and we do a lot of digitalization (pictures, movies), and of course regular backups, for which I use rsync. And we have three backup disks which are regularly changed. We are mostly running XP workstations, with a SAMBA server, but the backup disks are NTFS. Currently I am replacing the old 1 TB backups (which are nearly full) with 2 TB disks. And then I saw a problem re-occeur which I thought I had solved before. Not only that, but I also saw it vanish this time, rather than being solved, as I thought before. Take the example I gave already. rsync: recv_generator: failed to stat /mnt/tmp/backup/museum/adlib/adlib documenten voor/catalogus objecten in eigen bezit/artikelen, algemeen/8467 wehrpa\#303\#237.doc: Invalid or incomplete multibyte or wide character (84) rsync: recv_generator: failed to stat /mnt/tmp/backup/museum/documenten/bestuur/archief, niet geschoond/publiciteit/idee\#343\#253n website.doc: Invalid or incomplete multibyte or wide character (84) Behind the backslashes you see a numerical code, which is clearly indicating a locality problem; in fact the file names are 8467 wehrpaß.doc (with a German sz in it), and ideeën website.doc, with a Dutch e-trema in it. Neither of these is a problem in Linux, Samba, Windows XP, or NTFS. So I am pretty sure it _is_ a locality problem, and I had it before, when I installed the 1TB disks, and I _thought_ I had solved it. But now I also discovered that the problem vanishes by itself, when you re-run rsync. If you have this ideeën file, it is rejected, and it is _not_ saved, but if you run rsync again, the message does not appear, and the file _is_ saved. This must have led me astray the previous time in thinking that I had solved the problem, and it may also be responsible for some of the options I chose and which apparently were succesful. So my question is now: how come a file is rejected the first time around, and simply accepted the second time, _without_ any change? Well, as a system manager I can now be happy, for the problem goes away, but I am still interested in the mechanism behind it. My C knowledge is a bit rusty, so I am not really planning to go through the source code myself :-) -- joop gerritse Mühlenstraße 11 D-47546 Kalkar-Wissel Germany +49-2824-971487 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync to a Remote NAS
Hi, I don't think rsync natively supports samba shared volumes. You should probably start by mounting your shared volume, using mount -t smbfs, then sync. See http://users.softlab.ece.ntua.gr/~ttsiod/backup.html Greg On Wed, Apr 11, 2012 at 11:31 PM, Chris Arnold carn...@electrichendrix.comwrote: Forgive me if this has been addressed here before. We have a remote office that we need to backup to our NAS. We have a site to site certificate VPN. The remote site has over 51gb that needs to be backed up to our NAS over that VPN. I have tried this command: rsync --verbose --progress --stats --compress --recursive --times --perms --links --delete /Share/* / smb://192.168.123.6/Backup/EdensLandCorp and it just sits there and appears to do nothing. Does rsync make a tarball first and then put it where it is told to put it or does it just copy the files/folders over? Maybe it is the smb://xx.xx.xx.xx/whatever that is breaking it..the bottom line is i need to copy/rsync a directory to a remote server through a VPN. How is this accomplished? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem syncing to Netapp (rsync: failed to set times on...)
PPS. Yet another (a bit ugly) workaround, regarding how often the job is done : try the options --modify-window=N (N in seconds), to allow timestamp differences, or/and --size-only (comparing on filesize only). Greg On Mon, Apr 9, 2012 at 10:32 PM, Joachim Otahal (privat) j...@gmx.netwrote: I know from a lot of NAS boxes that they tend to use their internal time to stamp files instead of the time given by a copy job. The easiest way to test is to deliberately set the time off by a few hours on the box you monted the stuff on, the NAS and netapp (or the Server accessing the netapp) and create a file from your mount point, and check whether the time is right. cifs isn't the fastest way in unix environments, and samba uses _quite_ an amount of CPU power if you are above 50 MB/s. If in any way possible you should do it more directly, having an in-between box in the network causes (if -c is used) that all files are read from both boxes over the network just for the checksum, hence the bad performance. These Netapps, are they pure storage and the server using them is either windows or linux? If yes: Put rsync directly on those servers. This also applies to most NAS boxes I know, they offer rsync directly, most of the time as server. Joachim billdorr...@pgatourhq.com schrieb: Hey folks. I have a machine that I use as an intermediary to rsync between a NAS in another building (the users there have poor bandwidth and need local storage) and the our Netapp located in our Datacenter. I get lots of the rsync: failed to set times on... errors, and files which have already been transferred just try to sync again anyway. These are the mount options that I use for both sides: mount -t cifs -o credentials=/root/.synccreds //nasdevice/folder /nas mount -t cifs -o credentials=/root/.synccreds //netapp/folder /netapp The .synccreds file has the credentials of an Active Directory Domain Admin account, which has Full Control on both the NAS and the Netapp. Here is the command that I run to do the rsync: rsync -rvt --delete --progress /nas/ /netapp/ Running rsync with -i shows that the files are transferring because of timestamp differences. I tried the -c option in place of the -t, but the server doing the sync just hung there for literally two days without anything transferring and no output. I realize that the -c option is slower, but yikes! Any thoughts? Suggestions? Thanks, Bill D. Bill Dorrian Network Administrator Desk: 904-273-7625 Cell: 904-859-9471 THE PLAYERS begins May 10 on Golf's Greatest Stadium. For more information visit PGATOUR.COM/theplayers. The information contained in this transmission and any attachments may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: An alternative behavior for symlink
Hello All, I've been googling a lot but I can't find any suitable solution or advice, so I'm asking you. I'm trying do find a way to sync two distant folders containing symlinks pointing to files outside the synced folders BUT with different target names. Here is an example: ./dir1: (source) file -- ../data/file ./dir2: (destination) file -- ../share/other_file The goal would be to update 'share/other_file' with the contents of 'data/file', without modifying the existing 'dir2/file' symlink. I couldn't find any combination of option (--copy-unsafe-links, -l, -L, etc.): either 'dir2/file' symlink is deleted and replaced with a real file, or 'dir2/file' is replaced by a symlink pointing to '../data/file'... that doesn't exist in destination tree. I would really appreciate if anyone had any idea to solve this case. Thanks, Greg (Paris) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: An alternative behavior for symlink
Thank you for your answer, too bad there's no way, but it's understandable.
Have a nice day too,
Greg
Voelker, Bernhard wrote:
Greg Deback (rsync) wrote:
Hello All,
I've been googling a lot but I can't find any suitable solution or
advice, so I'm asking you.
I'm trying do find a way to sync two distant folders containing
symlinks pointing to files outside the synced folders BUT with
different target names.
Here is an example:
./dir1: (source)
file -- ../data/file
./dir2: (destination)
file -- ../share/other_file
The goal would be to update 'share/other_file' with the contents of
'data/file', without modifying the existing 'dir2/file' symlink. I
couldn't find any combination of option (--copy-unsafe-links, -l, -L,
etc.): either 'dir2/file' symlink is deleted and replaced with a real
file, or 'dir2/file' is replaced by a symlink pointing to
'../data/file'... that doesn't exist in destination tree.
I would really appreciate if anyone had any idea to solve this case.
There's no way. If you want to sync the content of ../data/file to
../share/other_file, then you have to do this explicitly.
rsync -aHxi ${source}/../data/file ${destination}/../share/other_file
Have a nice day,
Berny
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
