Re: --delete flag not working on cygwin rsync
On Fri, 2009-08-14 at 07:24 -0700, Wayne Davison wrote: > On Tue, Aug 04, 2009 at 03:10:20PM -0400, Sharad K wrote: > > Everything works fine except the --delete flag. > > Does rsync output a warning about an I/O error, saying it is disabling > deletes? You can use --force if you're sure you want rsync to override > that. --ignore-errors, not --force. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 3.0.4 under cygwin: problems with --exclude
On Tue, 2009-08-18 at 16:58 +0200, Alessandro Toso wrote: > I'm trying to backup a disk using rsync but I need to exclude some folder. > I'm using rsync 3.0.4 under cygwin on a winxp machine. > The script that I use is: > > # # > SRCDIR=/cygdrive/d > LOGDIR=/cygdrive/f > BCKDIR=/cygdrive/f/Backup > > rsync -auv --progress --delete --delete-excluded --log-file=$LOGDIR/`date > +%Y%m%d`.log --exclude from='/cygdrive/f/.bak_exclude' $SRCDIR/ $BCKDIR > # # > > and my file where there is the list of excluded folders looks like: > > # # > - /cygdrive/d/Software/ > - /cygdrive/d/Intel/ > - /cygdrive/d/Sysprep2003/ > - /cygdrive/d/RECYCLER/ > # # > > The problem is that rsync is not skipping these folders. Exclude patterns with a leading slash are interpreted from the root of the transfer, in this case /cygdrive/d/ . So you should not include that prefix in your patterns. See "ANCHORING INCLUDE/EXCLUDE PATTERNS" in the rsync man page for more details. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: support for ext-attrs on non-supporting FS's
On Mon, 2009-08-24 at 09:43 -0700, Linda A. Walsh wrote: > Maybe rsync could support extattrs on non-supporting fs's with a > ..xattr file stored > at the same place as the : > If a user used a switch to emulate 'xattrs', then on a fs that > didn't support real .xattrs one > could still ACL and other .xattr info. If a user ran rsync w/o that > switch such files > would be treated as 'normal files' -- allowing copying as normal files > between other > targets, but allowing ACL restoration on some final target by re using > the --emulate > switch, where it would detect the files on the source, and support on a > target, and > set ACL's appropriately? > > Just a thought? Doable or reasonable? I think the xattr emulation would be more appropriately done in a virtual filesystem to be layered over the real one (e.g., with FUSE) than in rsync. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: support for ext-attrs on non-supporting FS's
On Mon, 2009-08-24 at 22:09 -0700, Linda A. Walsh wrote: > FUSE runs on windows? I took a quick look and found this "FUSE windows port": http://fuse4win.4host.ru/ And Dokan, a user-mode filesystem layer for Windows which doesn't appear to be directly compatible with FUSE (but I could be mistaken): http://dokan-dev.net/ > But extended attributes? Most of the gnu utils aren't there yet, There are getfattr and setfattr for manipulating extended attributes from the command line. Were you thinking of something different? -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: content of file
On Thu, 2009-08-27 at 22:57 -0400, Mag Gam wrote: > Is it possible to stream the content of a file using rsync to stdout > instead of placing it into a file? No. Consider rdiff, which lets you call each of the three steps of the delta-transfer algorithm from a script. Or if you explain your use case further, I might have more ideas. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync proxy
On Wed, 2009-08-26 at 22:12 +1200, Nathan Ward wrote: > I'm trying to write an rsync 'proxy' of sorts. The plan is that my > code runs on two machines (one 'client' and one 'server') and each > piece of code executes a copy of rsync, and copies move in one > direction (server -> client). > > I have been able to run rsync on the 'server' end by calling it with -- > server --sender and so on. On the client end I have rsync call my code > with -e "my_code", however I am trying to make it so that on the > 'client' end, I can have my code call rsync, instead of the other way > around. > > When I call --server on the 'client' end, rsync seems to handshake OK, > but I get buffer overflow errors: > > ERROR: buffer overflow in recv_rules [sender] > rsync error: error allocating core memory buffers (code 22) at / > SourceCache/rsync/rsync-35.2/rsync/util.c(121) [sender=2.6.9] > > > The above is sent from the 'server' to the 'client'. > > Before I go delving in to the code, is --server supposed to be used in > this way? I am basically attempting to join two rsync processes both > running --server, but only one running --sender. No, that will not work. The rsync protocol requires one client and one server. See https://bugzilla.samba.org/show_bug.cgi?id=5220 for some ideas on how to call an rsync client from your code and get it to use your existing connection. > The background here is I'm writing a backup tool and need to do a few > more things than rsync can do alone, but there's no point replicating > the stuff that rsync *can* do. I also don't want to use the rsync > daemon, nor do I want to have a user account that is remotely > accessible in order to get rsync over ssh going. Yes I know there are > solutions for parts of this, but I want to write this tool all the same. Indeed, there may be better solutions for the whole thing if you explain your use case further. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: unpack_smb_acl unknown error, + rsync maintainability
On Mon, 2009-08-31 at 16:59 -0500, Mike Bombich wrote: > Regardless, rsync shouldn't be converting uuid to uid/gid for Mac OS > X, it's unnecessary. This is really a question of whether Mac users expect users/groups to be preserved by uid/gid or by uuid. Are you saying it makes more sense to preserve by uuid? What would --numeric-ids do? More broadly, I don't think it's a scalable approach to continue adding ad-hoc workarounds to rsync for the quirks of various filesystems. It would be better to have a filesystem-specific component that exposes all the metadata it knows about as named attributes (gvfs-info does something like this) and let the core file-copy tool focus on efficiently transferring and flexibly manipulating that representation. But such extensible design seems to be uncharacteristic of the current rsync codebase. Maybe it would be better to pursue this idea in a new tool along the lines of rZync and superlifter. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Old versions of lib/addrinfo.h missing from rsync repo
Wayne, Today, to investigate an issue on RHEL 5.2, I checked out rsync 2.6.8 from the rsync repository and attempted to build it. The build failed because the tagged tree did not contain lib/addrinfo.h, which is included by rsync.h. On further investigation, the 2007-10-24 snapshot of the CVS repository you sent me contains revisions 1.1 through 1.11 of lib/addrinfo.h with the release-2-6-8 tag on 1.9, but those eleven revisions are missing from the current CVS repository and the git repository (which I assume was generated from the CVS). Why were they dropped from the development history? There doesn't appear to be a copyright issue: the file bears a BSD license. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsyncd always compression
On Tue, 2009-09-01 at 14:11 +, Andrew Gideon wrote: > On Mon, 31 Aug 2009 13:37:16 -0700, Wayne Davison wrote: > > >> Anyone knows a trick that the server only answer if the client is use > >> the compression? > > > > This is not currently possible. > > What if rsync-path is set to a little script that only accepts the > connection (and exec()s the real rsync binary) if compression is set? The question was about an rsync daemon. But something similar is possible: the daemon can be configured with a "pre-xfer exec" script that looks at the RSYNC_ARG# environment variables and exits unsuccessfully if compression is not enabled. The downside is that, like rrsync, the script has to reimplement rsync argument parsing and the enforcement is only as good as the argument parsing. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Old versions of lib/addrinfo.h missing from rsync repo
On Tue, 2009-09-01 at 23:09 -0700, Wayne Davison wrote: > On Wed, Sep 02, 2009 at 12:32:45AM -0400, Matt McCutchen wrote: > > Why were they dropped from the development history? There doesn't > > appear to be a copyright issue: the file bears a BSD license. > > It used a 3-clause BSD license, which is incompatible with the GPL. I think you're mistaken. Here is the text of the license (I have checked that it is the same in all the revisions in question): /* * Copyright (C) 1995, 1996, 1997, 1998, and 1999 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright *notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright *notice, this list of conditions and the following disclaimer in the *documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors *may be used to endorse or promote products derived from this software *without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ This looks like the Modified BSD License in the GNU license list: http://www.gnu.org/licenses/license-list.html#ModifiedBSD which is stated to be compatible with the GPL. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Old versions of lib/addrinfo.h missing from rsync repo
On Tue, 2009-09-01 at 23:09 -0700, Wayne Davison wrote: > The > incompatible code was expunged from CVS and was never imported into git. Two further remarks: - You didn't expunge http://rsync.samba.org/ftp/rsync/src/rsync-2.6.8.tar.gz . - You could have saved me some head-scratching by redacting lib/addrinfo.h to contain only an #error describing the situation, rather than removing the file altogether. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: unpack_smb_acl unknown error, + rsync maintainability
On Tue, 2009-09-01 at 09:09 -0500, Mike Bombich wrote: > Rsync shouldn't be converting UUID to uid/gid *for ACL entries* > (because Mac OS X stores those ACEs with a reference to a UUID, not a > uid/gid). For file ownership, etc., it should maintain its current > behavior. My point is that performing or not performing the conversion may lead to different results if the uuid-id mapping is different on the two machines involved. So the question is not just one of efficiency, but rather which result users are more likely to want. I thought that you, as a Mac user, might be able to answer that. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using rsync for buidling Oracle standbys.
On Thu, 2009-09-03 at 10:25 -0700, Saibabu Devabhaktuni wrote: > We currently use rsync to create an Oracle standby on a target box > from an existing standby by copying all the datafiles while the source > standby is in recovery status. We are occasionally running into > datafile corruptions being reported by oracle when it is recovering > the new standby. Oracle support says they don't recommend copying > files while the changes being applied to them unless the database is > put in backup mode (which is not possible for us). > > We are using "rsync -av --password-file=pass.secret > source_data_file_name ora...@target::ORCL/" > > We are planning to use "rsync -av --password-file=pass.secret > --block-size=8192 source_data_file_name ora...@target::ORCL/" > > This is because, we are using 8k block size for Oracle and hence the > blocks will be copied in full by rsync to avoid fractured block > copies. Am I correct in this assumption by using "--block-size" > option? No. --block-size sets the block size for the delta-transfer algorithm; rsync does not provide a way to set the size of its read(2) calls. And even if it did, it would appear from the following message that read(2) calls are not atomic on Linux: http://marc.info/?l=linux-kernel&m=107375454908544 I would recommend looking into a block-device-level snapshot facility such as LVM snapshots. (Note: a similar issue was discussed in this thread: http://lists.samba.org/archive/rsync/2009-May/023270.html but the non-atomicity of read(2) calls was not considered.) -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [manpage] Error in French manpage for "--checksum-seed" description
On Sat, 2009-09-05 at 14:25 +0200, Gilles-Alexandre wrote: > I think IMHO that the description of the option "--checksum-seed" > should be : > > "--checksum-seed=NUM spécifie le grain de sable pour la somme de > contrôle" The rsync project does not maintain a French manpage. Please report this to the third party that maintains the manpage in question. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: how to use rsync installed on network
On Mon, 2009-09-07 at 23:22 +0100, e-letter wrote: > Scenario: file.txt on local machine, to be transferred to remote host. > Remote host has rsync installed. Reading the manual I the examples > describe scenarios where rsync is installed on the local machine. Is > rsync suitable to use when installed on the remote host? To do a remote transfer, rsync needs to be installed on both the local host and the remote host. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: originate sync from the daemon server
On Thu, 2009-09-10 at 08:42 -0500, Kevin Kramer wrote: > is there anything special to do this from the daemon server. I've setup > the /etc/rsyncd.conf with some filesystems and I would rather originate > (control) my rsyncs from this server and not from the hosts that have > the data I want. ie. I want to pull not push. Then you don't need to run an rsync daemon on the machine that will be collecting the data. Just have that machine run ordinary rsync client commands at the desired interval. > what should my rsync line look like in a cron or command line > > /usr/local/bin/rsync -axS --delete --stats servername:/var/www/ www It looks like that command is trying to copy directly to the daemon. That would have to be written: /usr/local/bin/rsync -axS --delete --stats servername:/var/www/ localhost::www or: /usr/local/bin/rsync -axS --delete --stats servername:/var/www/ rsync://localhost/www But that's the case known as "two remote endpoints" (even though the daemon happens to be on the same machine), which the rsync client currently doesn't support. It needs one endpoint to be in the local filesystem. So you could bypass the daemon and do: /usr/local/bin/rsync -axS --delete --stats servername:/var/www/ /snaps/www The daemon options will need to be translated into appropriate options to, or scripting around, this rsync command. E.g., have a script that runs rsync with --numeric-ids and then your "post-xfer exec" command. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync deleted all my important docs...
On Sat, 2009-09-12 at 18:22 -0300, Carlos Carvalho wrote: > Sanjay Acharya (svachary...@gmail.com) wrote on 11 September 2009 11:02: > >How is rsync supposed to behave when I give the --delete-excluded? > > >From the manual for the --delete option: > >If the sending side detects any I/O errors, then the deletion of >any files at the destination will be automatically disabled. >This is to prevent temporary filesystem failures (such as NFS >errors) on the sending side causing a massive deletion of files >on the destination. Unfortunately, this safety feature is partially disabled when rsync uses protocol 30 (default for rsync 3.0.*) with incremental recursion and --delete-during, because it caused a deadlock that could not be fixed without breaking protocol compatibility; see commit deea1f70bd67c6cc9249def9164f5e61e786e845 . So Sanjay, if you're having problems with IO errors, I would suggest disabling incremental recursion with --no-i-r or using the current development rsync (which supports protocol 31, which has the issue resolved) on both sides. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: time variable throttling rsync traffic
On Sun, 2009-09-13 at 21:02 -0400, Eric S. Johansson wrote: > I am using rsync to back up across a VPN. Unfortunately, every so often the > home office miscreants drop a big block of data into the backup and that > particular backup cycle takes many hours. These same people also complain > when > net pipe is filled during the day. > > What I need is an ability to change the bwlimit based on time of day. Any > suggestions as to how to do this? How about the suggestions you were given on the rsnapshot list? http://sourceforge.net/mailarchive/forum.php?thread_name=1252620930.2041.30.camel%40localhost&forum_name=rsnapshot-discuss -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync deleted all my important docs...
On Mon, 2009-09-14 at 10:30 -0300, Carlos Carvalho wrote:
> Matt McCutchen (m...@mattmccutchen.net) wrote on 13 September 2009 16:53:
> >On Sat, 2009-09-12 at 18:22 -0300, Carlos Carvalho wrote:
> >> Sanjay Acharya (svachary...@gmail.com) wrote on 11 September 2009 11:02:
> >> >How is rsync supposed to behave when I give the --delete-excluded?
> >>
> >> >From the manual for the --delete option:
> >>
> >>If the sending side detects any I/O errors, then the deletion of
> >>any files at the destination will be automatically disabled.
> >>This is to prevent temporary filesystem failures (such as NFS
> >>errors) on the sending side causing a massive deletion of files
> >>on the destination.
> >
> >Unfortunately, this safety feature is partially disabled when rsync uses
> >protocol 30 (default for rsync 3.0.*) with incremental recursion and
> >--delete-during, because it caused a deadlock that could not be fixed
> >without breaking protocol compatibility; see commit
> >deea1f70bd67c6cc9249def9164f5e61e786e845 .
>
> Searching in the gitweb I didn't find this commit. How can I discover
> it?
http://gitweb.samba.org/?p=rsync.git;a=commit;h=deea1f70bd67c6cc9249def9164f5e61e786e845
That commit was the one I remembered seeing on the 3.0.x branch; it
removed the safety check once the deadlock was discovered. I did a "git
log -Scheck_for_io_err" on the trunk, finding that the feature was added
in the same problematic form and then changed to the working version for
protocol 31:
commit 8b3e60523aa3d86583a1df54c81c9d480ea19740
Author: Wayne Davison
Date: Sun Aug 31 09:03:50 2008 -0700
Improved the fix that ensures that the generator gets notified about an
I/O error for the incremental directory that generated the error. The
PROTOCOL_VERSION was bumped to 31 to implement this.
commit f303b749f2843433c9acd8218a4b9096d0d1bb8d
Author: Wayne Davison
Date: Sat Aug 2 09:14:36 2008 -0700
Added logic to the receiving side to ensure that the --delete-during
code will not delete in a directory prior to receiving an I/O error
for that directory (or not receiving it, as the case may be).
> Also, I see no mention of this in the changelog.
Indeed, there is none. There probably should be one.
> What's the exit status in this case? From what Sanjay says:
>
> Sanjay Acharya (svachary...@gmail.com) wrote on 11 September 2009 11:02:
> >rsync: readlink_stat("/mnt/raid-backup/Documents/Finance") failed:
> >Input/output error (5)
>
> it appears to be 5. Could it be 23 as well? 23 is very frequent and
> usually not a problem.
23 means that the run completed with at least one nonfatal error that
may affect its correctness. Such errors can range from inability to set
the mtime of a destination directory owned by someone else to
nonexistence of an entire source argument, and the severity of a
particular error may depend on the use case. So I would hesitate to
make generalizations.
The "Input/output error (5)" is an errno, which has no relationship to
rsync exit codes. The error you cited would raise code 23, and
moreover, since it can result in the omission of source files from the
file list, it would set IOERR_GENERAL to suppress deletion.
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Status of running rsync
On Tue, 2009-09-15 at 08:37 +0200, Fabian Cenedese wrote: > Is there a way to get the status of a running rsync process? AFAIK, rsync has never had such a feature. One thing you can do is temporarily attach strace. > I believe > it has to be done by sending a signal, but I can't find it anymore. Maybe you're thinking of dd, which prints its status when it receives SIGUSR1. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Proposed tweaks
On Mon, 2009-09-14 at 21:13 -0400, Lee Winter wrote: > The purpose of this note is to inquire about your collective interest > in optimizing rsync for certain uses, particularly atomic, > unidirectional transfers with few or single writer and many, often > very many, readers. > > After studying the requirements and the documentation on samba.org I > have reached the tentative conclusion that serious improvements can be > obtained with a few small tweaks and one medium-sized internal change. Would you care to elaborate to attract more comments? -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Proposed tweaks
Lee, Both of your proposals have been discussed before (see below), but neither has been taken very far because they would both involve large changes to rsync. On Wed, 2009-09-16 at 01:16 -0400, Lee Winter wrote: > A simplistic approach would be to add a file-list option able to > create-and-save a new file list or load-and-use an existing file list. > This would permit a newly updated mirror to perform the file list > generation once per update rather than once per client. See: https://bugzilla.samba.org/show_bug.cgi?id=3579 A first step in this direction is the maintained patch "db.diff", which caches checksums of entire source files but does not (yet) cache the file list itself. > But in an severely asymmetrical environment where the goal is minimal > load on the sender, it would be better to swap the roles by using the > sender's newer version as the "basis file" and let each receiver chew > through its comparisons rather than having the sender perform the > comparisons for all of the receivers. This is a good idea that has been discussed before and is implemented in another tool called "zsync". Wayne noted some of its drawbacks here: http://markmail.org/message/pt354zo4njgmupjj Perhaps you could respond to his concerns. > But, while I did see some traffic about the > conflict between delta processing and compressed files, I never > located a resolution or summary of the issue. Does the current > version of rsync actually provide much benefit for compressed files? Not for typical compression formats, where a local change to the input changes the output from that point on. If compression would preserve locality of changes, then delta transfer would work, but locality is somewhat at odds with effective compression. There used to be a patch "gzip-rsyncable.diff" that modified gzip for better locality, but I don't know if it's maintained any more. The other approach is to delta-transfer the uncompressed content: https://bugzilla.samba.org/show_bug.cgi?id=2296 -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Can I get rsync to delete missing files first?
On Thu, 2009-09-17 at 14:41 +1200, Allistar wrote: > I am backing up a remove partition which is 38Gb onto a local partition > which is 40Gb. Rsync is complaining that I have run out of disk space on > the local partition, and I can tell that this would not happen if local > files that are missing in the remote system were deleted BEFORE copying > down new files. > > I note there is a --delete-after option, but is there a --delete-before > option? Yes, there is! -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Can I get rsync to delete missing files first?
On Thu, 2009-09-17 at 14:41 +1200, Allistar wrote: > I am backing up a remove partition which is 38Gb onto a local partition > which is 40Gb. Rsync is complaining that I have run out of disk space on > the local partition, and I can tell that this would not happen if local > files that are missing in the remote system were deleted BEFORE copying > down new files. > > I note there is a --delete-after option, but is there a --delete-before > option? If not, can I simulate this somehow? > > This is rsync version 2.6.2 protocol version 28 Oops, I didn't notice the really old rsync version. In rsync 2.6.2, just use --delete to "delete before". If you are doing that and are still running out of disk space, something else may be wrong. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync deletes target files after source drive failure
On Thu, 2009-09-17 at 20:59 +0200, Enno Middelberg wrote: > Here the two directories which couldn't be read have been deleted from > the target directory (ie, the files contained in them were deleted, > the directories did still exist, if I remember correctly). If I > understand the man page correctly, rsync should not delete files from > the target directory when an error occurs. This can be overridden with > the "--ignore-errors" command line switch, but this switch has > definitely not been used. Maybe you are hitting this issue: https://bugzilla.samba.org/show_bug.cgi?id=6719 -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: "quick check" failed
On Thu, 2009-09-17 at 12:46 +0200, Harald Dunkel wrote: > We ran 2 mirror jobs using "rsync -av remote:/from /to" > of appr. 700 GByte. Problem: On the second run about 10 or 15 > files were copied again, even though we are very sure that there > was no service running which could have changed the files on > remote:/from or on /to. Esp. Samba and NFS were off. > > How comes? How should I know? Perhaps rsync failed to preserve mtimes, or they were subsequently changed? If you pass -i, rsync will tell you why it is transferring each file. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync finds files to sync which don't exist and then crashes
On Sat, 2009-09-26 at 14:45 -0700, stefaan wrote:
> I'm running the following script for backup:
> SHORTHOST=`hostname | awk 'BEGIN { FS="." } { print $1 }'`
> cd $HOME || exit 1
> rsync --ignore-errors --max-size=50m -v --delete --delete-excluded
> --log-file=/tmp/backup.log --exclude-from=$HOME/.rsync/exclude \
> --timeout=3 -az \
> $HOME atticserve:/mnt/store/backup/$SHORTHOST
>
> At some point it tries to sync the following file:
> stefaan/Pictures/2009-08-29/P1000908.JPG
>
> This file does not exist, although it used to.
That's really strange and smells of flakiness with the filesystem. How
specifically did you check that the file does not exist: ls? Does
"rsync --list-only" show the file? It may help to run the sending side
of rsync under strace to see its exact interaction with the filesystem.
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --compare-dest and empty directories
On Fri, 2009-10-02 at 03:21 -0500, Albert Chin wrote: > With rsync-3.0.6 on a Solaris 10/x86 system: > $ mkdir /opt/s/c > $ cd /opt/s/c > $ gtar zxf rsync-3.0.6.tar.gz > $ mkdir /opt/s/o > $ rsync -aHv rsync-3.0.6 /opt/s/o > $ mkdir /opt/s/o2 > $ cd /opt/s/o/rsync-3.0.6 > $ echo foo >configure > $ rsync -aHv --compare-dest=/opt/s/c /opt/s/o/rsync-3.0.6 /opt/s/o2 > $ cd /opt/s/o2 > $ find . > ./rsync-3.0.6 > ./rsync-3.0.6/doc > ./rsync-3.0.6/lib > ./rsync-3.0.6/packaging > ./rsync-3.0.6/packaging/bin > ./rsync-3.0.6/packaging/lsb > ./rsync-3.0.6/packaging/solaris > ./rsync-3.0.6/popt > ./rsync-3.0.6/support > ./rsync-3.0.6/testhelp > ./rsync-3.0.6/testsuite > ./rsync-3.0.6/zlib > ./rsync-3.0.6/configure > > Is there a way, with --compare-dest, to perform the transfer without > creating empty directories? What I want is: > $ find . > ./rsync-3.0.6 > ./rsync-3.0.6/configure No. See: http://lists.samba.org/archive/rsync/2009-January/022488.html http://lists.samba.org/archive/rsync/2005-October/013776.html -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Exclude directory from --delete
On Sat, 2009-10-03 at 15:47 -0700, J. Ellis wrote: > I have the following 3 commands which are supposed to run every night > to back up my web server: > > rsync -avve ssh --numeric-ids --delete --ignore-errors > myu...@myisp.com:/usr/home/myuser /Volumes/Downloads > rsync -avve ssh --copy-links --numeric-ids --delete --ignore-errors > myu...@myisp.com:/usr/home/myuser/mail_boxes > /Volumes/Downloads/myuser/mail_boxesBU > rsync -avve ssh --copy-links --numeric-ids --delete --ignore-errors > myu...@myisp.com:/usr/home/myuser/mail_boxes > /Volumes/Downloads/myuser/mail_boxesBU > > The way this works is that there are two symlinks on the server to the > mail_boxes and public_html directories. I didn’t want all the symlinks > within those directories to be followed, so these I backup separately > to mail_boxesBU and public_htmlBU respectively. There happens to be a trick for "copying a few dirlinks" without copying all of them: use --relative and specify an additional source argument that goes through each symlink with a trailing slash. E.g.: rsync -avve ssh --numeric-ids --delete --ignore-errors -R \ myu...@isp.com:/usr/home/./myuser \ myu...@isp.com:/usr/home/./myuser/mail_boxes/ \ myu...@isp.com:/usr/home/./myuser/public_html/ \ /Volumes/Downloads For more about this, see: http://lists.samba.org/archive/rsync/2006-February/014838.html > Unfortunately, when the first command is run, both are deleted from > the local volume, and then must be completely downloaded from scratch > each night after the deletion, instead of simply being updated. This > doesn’t always work for some reason, and the next morning one or both > of the directories are just gone. > > Is there some way of excluding the mail_boxesBU and public_htmlBU > directories from --delete? If you want to keep your existing approach, yes, you can just use --exclude='/myuser/mail_boxesBU' and --exclude='/myuser/public_htmlBU'. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Exclude directory from --delete
On Sat, 2009-10-03 at 21:03 -0700, J. Ellis wrote: > So let me see if I understand correctly. First, are the \ characters > followed by a return, or are they all on a single line? In other words, is > this entered in the terminal as one command, or does it need to be broken up > somehow? The \ is a notation for line continuation recognized by bash, among other programs. So you can either type the command with backslashes and returns as I gave it or put everything on a single line without backslashes. > And then each of the symlinks for the directories specified will be treated > as symlinks, Directories, I think you mean. > but any symlinks below them will still be treated as symlinks? Correct. It's magic. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Exclude directory from --delete
On Sat, 2009-10-03 at 22:06 -0700, Jeffrey Ellis wrote: > Hm. I've tried all of the following: > > rsync -avve ssh --numeric-ids --delete --ignore-errors -R > myu...@myisp.com:/usr/home/./myuser > myu...@myisp.com:/usr/home/./myuser/mail_boxes/ > myu...@myisp.com:/usr/home/./myuser/public_html/ /Volumes/Downloads > > (no returns, only spaces here. Line breaks here are actually a single > space). > > rsync -avve ssh --numeric-ids --delete --ignore-errors -R > \myu...@myisp.com:/usr/home/./myuser > \myu...@myisp.com:/usr/home/./myuser/mail_boxes/ > \myu...@myisp.com:/usr/home/./myuser/public_html/ \/Volumes/Downloads > > (again, no returns. Line breaks here are a single space, followed by a \). > > rsync -avve ssh --numeric-ids --delete --ignore-errors -R \ > myu...@myisp.com:/usr/home/./myuser \ > myu...@myisp.com:/usr/home/./myuser/mail_boxes/ \ > myu...@myisp.com:/usr/home/./myuser/public_html/ \ /Volumes/Downloads > > (no returns. Line breaks are a single space, followed by a \, then another > single space). > > rsync -avve ssh --numeric-ids --delete --ignore-errors -R \ > myu...@myisp.com:/usr/home/./myuser \ > myu...@myisp.com:/usr/home/./myuser/mail_boxes/ \ > myu...@myisp.com:/usr/home/./myuser/public_html/ \ > /Volumes/Downloads > > (These are a space followed by a \ followed by a return. I'd actually prefer > the single line). > > I'm definitely in the bash shell. > > These all come up with errors, so I must be doing something wrong. Your first and last attempts should have worked. What error do you get? -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Filter file: selected directories only
On Sun, 2009-10-04 at 00:29 +0400, rs...@xbit.ru wrote: > I'm trying to create a filter file for rsync what saying which dirs to > copy. For example i need only /lib and /var/lib dirs from remote server. > I say: > rsync -f ". /tmp/rsync_dirs" --dry-run --verbose "remote:/" > "/local_dir/". > > File /tmp/rsync_dirs looks like: > + /lib/ > + /var/ > -! /var/lib/* > - /* > > This doesnt work: it copies all /var/ dir except contents of /var/lib/. > What's wrong? Your third rule excludes all files that don't match "/var/lib/*", and /var/lib is such a file since it has no third path component to match the "*". Hence the behavior you're seeing. Instead, try this approach of including the /var/lib dir and then excluding everything else at the /var/* level: + /lib/ + /var/ - /* + /var/lib/ - /var/* In practice, I have never found a legitimate use for "!" patterns and believe they are a misfeature. From the version-control history, it appears that the only use Wayne had in mind for them was to replace the traditional --include='*/' --exclude='*' (or --filter='S */' --filter='H *') pattern, but I really don't see what was wrong with that pattern, as it is consistent with the spirit of all other filter usage I have seen. Wayne, comments? -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --keep-dirlinks --delete erroneously deletes symlinks to directories
On Sat, 2009-10-03 at 12:59 +0200, Roland Koebler wrote: > the combination of --keep-dirlinks and --delete (--delete-during, > --delete-delay) erroneously deletes symlinks to directories on > the receiver. > With --delete-before and --delete-after it works as expected. > > (rsync version 3.0.3 protocol version, on Debian Lenny) > > I don't think that this behaviour is intended. > > > Details/example: > > - Host 1 directory-structure: > drwx-- ./1 > -rwx-- ./1/file > drwx-- ./1/dir > -rwx-- ./1/dir/subfile > > - Host 2 directory-structure: > drwx-- ./2 > -rwx-- ./2/file > lrwxrwxrwx ./2/dir -> ../foo > drwx-- ./foo > -rwx-- ./foo/subfile > > - the data on both hosts is the same, so rsync shouldn't do anything > - mirroring Host 1 -> Host 2 correctly does nothing: > $ rsync -av --keep-dirlinks 1/ 2/ > sending incremental file list > > sent 94 bytes received 13 bytes 214.00 bytes/sec > total size is 0 speedup is 0.00 > > - mirroring Host 2 -> Host 1 correctly does nothing: > $ rsync -av --copy-dirlinks 2/ 1/ > sending incremental file list > > sent 94 bytes received 13 bytes 214.00 bytes/sec > total size is 0 speedup is 0.00 > > - --delete-after works correctly, and does nothing: > $ rsync -av --keep-dirlinks --delete-after 1/ 2/ > building file list ... done > > sent 89 bytes received 12 bytes 202.00 bytes/sec > total size is 0 speedup is 0.00 > > - --delete-before works correctly, and does nothing: > $ rsync -av --keep-dirlinks --delete-before 1/ 2/ > building file list ... done > > sent 89 bytes received 12 bytes 202.00 bytes/sec > total size is 0 speedup is 0.00 > > - but --delete (or --delete-during / --delete-delay) erroneously deltes the > "dir -> ../foo"-symlink (and copies the data afterwards): > $ rsync -av --keep-dirlinks --delete 1/ 2/ > sending incremental file list > deleting dir > dir/ > dir/subfile > > sent 136 bytes received 35 bytes 342.00 bytes/sec > total size is 0 speedup is 0.00 > > Is this a bug, or did I miss something? It looks like a bug. I can reproduce it with rsync 3.0.3 but not with 3.0.6. Please upgrade. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: (Recent I/O change)
On Sat, 2009-10-03 at 09:41 -0500, Wayne Davison committed: > Change the msg pipe to use a real multiplexed IO mode > for the data that goes from the receiver to the generator. Wayne, Would you care to explain the impact of this change on the rsync I/O design in terms that I might understand? It looks like you merged two switch statements over possible message types, reducing code duplication, but that's all I can really tell. Incidentally, I find that the big chunk of files-from code is distracting in the middle of read_timeout. I would suggest moving it to a separate function. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: LVM snapshots vs. --link-dest
On Mon, 2009-09-28 at 14:18 +, Andrew Gideon wrote: > It assumes that --inplace > actually does what it sounds like it does: it modifies only the disk > pages of a file that have changed, as opposed to changing the entire file > or creating a new file or some other thing which causes the file to > "move" on disk. Yes, --inplace works that way: http://gitweb.samba.org/?p=rsync.git;a=blob;f=receiver.c;h=965bb3909dc75d6b4ba675e201a64549ab8ac8dc;hb=HEAD#l295 -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: gnu --target-dir, how to fake with rsync?
On Mon, 2009-10-05 at 12:47 -0400, Sanjeev Sharma wrote: > Is this in rsync and I just can't find it, or if not, is there a way > to fake this OR, third possibility, is anyone working to add it? > > > For those who've never used this option - for the GNU cp & mv type > commands they have the > --target-dir="dir" > > option. It lets you put the destination directory as the first > argument to cp and mv instead of the last argument. > > They added the option to get cp & mv working well with xargs for > stuff like this: > > find ./ -type file -mindepth 3 -print0 | xargs -r0 cp > --target-dir=/home/sam/stuff/otherstuff/morestuff -- > > then xargs puts all of the files found by find after the -- The obvious approach is to use a wrapper script that moves its first argument to the end, like this: #!/bin/bash dest="$1" shift exec rsync "$@" "$dest" But if you're dynamically generating a list of source files, it might be appropriate to use --files-from rather than passing them as arguments. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --copy-unsafe-links does not work for "double" symlinks
On Tue, 2009-10-06 at 01:47 +0200, Roland Koebler wrote:
> as documented, if you use rsync with --copy-unsafe-links, and copy a
> directory with a symlink pointing outside of the copied tree, the
> referent of the symlink is copied.
>
> Now, assume that the directory contains a symlink, which points to a
> (2nd) directory outside of the copied tree.
> If this (2nd) directory contains another symlink, which also points outside
> of the copied tree, rsync unfortunately only copies this symlink and *not*
> the referent.
> Example:
>
> - host 1 directory-structure ("empty"):
> drwx-- ./1
>
> - host 2 directory-structure:
> drwx-- ./2
> -rw--- ./2/file
> lrwxrwxrwx ./2/dir -> ../foo
> drwx-- ./foo
> -rw--- ./foo/foofile
> lrwxrwxrwx ./foo/bar -> ../bar
> drwx-- ./bar
> -rw--- ./bar/barfile
>
> - mirror host 2 to host 1:
> $ rsync -av --copy-unsafe-links 2/ 1/
> sending incremental file list
> ./
> file
> dir/
> dir/bar -> ../bar/
> dir/foofile
>
> sent 206 bytes received 60 bytes 532.00 bytes/sec
> total size is 7 speedup is 0.03
>
> Now, host 1 contains:
> drwx-- ./1
> -rw--- ./1/file
> drwx-- ./1/dir
> -rw--- ./1/dir/foofile
> lrwxrwxrws ./1/dir/bar -> ../bar
> "file", "dir" and "foofile" are ok; but "bar" is a dangling symlink, and
> "barfile" was not copied.
Ah, yes. The problem is that rsync is evaluating the safety of the
symlink it sees at "dir/bar" as if "dir" were a real directory, ini
which case the symlink would indeed be safe. To fix this, rsync would
need to remember the last component of the file-list path that was an
unsafe symlink and count only the path components after that point in
symlink safety checks.
I've made a first attempt at implementing this in the
"wip/copy-unsafe-links-double" branch of my rsync repository at:
https://mattmccutchen.net/rsync/rsync.git/
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Filter file: selected directories only
On Thu, 2009-10-08 at 03:17 +0400, rs...@xbit.ru wrote: > Thank you, it works. > > But if I want to delete only certain files or dirs from remote side? > > example: > > $ tree dirA/ > dirA/ > |-- dir2 > | `-- file3 > |-- file1 > `-- file2 > > $ tree dirB/ > dirB/ > |-- dir2 > | `-- file3 > |-- file1 > |-- file2 > `-- protect1 > > $ rsync -av --dry-run --delete-excluded -f ". filter" dirA/ dirB/ > building file list ... done > ./ > > sent 74 bytes received 26 bytes 200.00 bytes/sec > total size is 0 speedup is 0.00 > > $ cat filter > H * > P * > R dir2* > - dir2* > > I want to delete "dirB/dir2/" only. Not "dirB/protect1" or any other > file or dir in "dirB/". > What sort of filter I must create? "rm -r dirB/dir2"? I don't think I understand your point here. > Where can I find example filter files for complicated cases? There are a few examples scattered throughout the man page, but I don't know of any really good tutorial. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: empty quotes for rsync parameter
On Sun, 2009-10-11 at 00:37 -0700, Wolfram Volpi wrote:
> Why does inserting an empty quote ("") into a rsync command cause the current
> directory to be backed up?
>
> Not sure if this is a UNIX-syntax question or rsync-syntax question.
>
> Code:
> #!/bin/sh
>
> echo "\nspace between quotes has no effect; only music is backed up:"
> rsync -ilrtn " " /home/wolf/music /backups/backup1
Um, no, it either backs up the file in the current directory whose name
is a single space, or gives an error that such a file does not exist.
> echo "\nempty quotes backs up current directory and music:"
> rsync -ilrtn "" /home/wolf/music /backups/backup1
This appears to be a feature:
http://gitweb.samba.org/?p=rsync.git;a=blob;f=flist.c;h=a167c3b9fb15d591d3e82aa37fddaa4c8c9a3e6e;hb=HEAD#l2118
It makes perfect sense to me that the empty path, as a relative path
with zero components, would refer to the current directory. In fact,
old versions of *nix behaved that way, but current ones reject the empty
path:
http://www.kernel.org/doc/man-pages/online/pages/man7/path_resolution.7.html
http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap04.html#tag_04_11
So one could argue that rsync should reject it too. I don't care about
the issue personally.
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: gnu --target-dir, how to fake with rsync?
On Wed, 2009-10-07 at 04:40 -0700, Martin Scharrer wrote: > I agree with both above points. However, I would also vote for adding such > an option to rsync. > This should not be to difficult and would come in handy. I'm not convinced of the need for a --target-dir option. Do you have an example in which it would be significantly cleaner than --files-from? > About the mentioned use of the option --files-from: > Sanjeev's earlier example: > > find ./ -type file -mindepth 3 -print0 | xargs -r0 cp > --target-dir=/home/sam/stuff/otherstuff/morestuff - > > can be adapted to rsync like this: > > rsync -0 --files-from=<(find ./ -mindepth 3 -type f -print0) > /home/sam/stuff/otherstuff/morestuff > > if you are using the BASH shell. Otherwise you have to write the output of > 'find' into a temp file and use this for --files-from. No, you can use stdin, as the man page description of --files-from mentions: find ./ -mindepth 3 -type f -print0 \ | rsync -0 --files-from=- /home/sam/stuff/otherstuff/morestuff -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem with symlinks
On Sun, 2009-10-11 at 13:07 +0200, Thomas Ebert wrote: > rsync has problems with symlinks. First some background information: > I regularly back up my / and /home from my desktop PC and my netbook to > my Synology DiskStation (500GB) using rsync 3.0.6 with this script: > http://pastebin.com/m43d4e353 . This works without any error messages. > On the DS are also stored other files from two windows machines. > > The DS has a built in backup feature that unfortunately has problem with > hard links so I stripped down my script to directly run rsync 3.0.4 on > the DS: > > |# Backup source device/share/directory/etc > BACKUP_SOURCE=/volume1/thomas/ > > # Backup destination device/directory > BACKUP_TO_DEV=/volumeUSB1/usbshare/thomas_backup/ > > rsync -q -v -axH --no-inc-recursive --delete --exclude="@eaDir" > $BACKUP_SOURCE $BACKUP_TO_DEV| > > This handles the hard links (-H) but now I have problems with symlinks. > When running the script I receive a lot of error messages from this type: > rsync: > open(backup/sammy-root/2009-07-29/usr/share/pixmaps/cxassoc-cxoffice-0-application_x-crossover-exe.png) > > failed!!: No such file or directory (2) > rsync: open(backup/sammy-root/2009-07-29/usr/share/qt/mkspecs/default) > failed!!: Is a directory (21) > rsync: > open(backup/sammy-root/2009-07-29/usr/share/texmf-dist/scripts/context/stubs/unix/luatools) > > failed!!: No such file or directory (2) Standard rsync does not produce such output containing the string "failed!!". Are you using the DS's custom version of rsync? If so, we don't support it on this list and you should bug the manufacturer. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Filter file: selected directories only
On Fri, 2009-10-09 at 20:54 +0400, rs...@xbit.ru wrote: > Well, I must say that this must by bone by rsync. > That's because I'm creating small automation system, > part of it is rsync-based. > Yes, I can write "rm -r dirB/dir2", but this will break consitency > of the whole system and removes some features that I need. Details, please? I know you're looking for a solution to some more general problem, but I can only help you to the extent that you tell me what that problem is. If you just want to delete something using rsync, look into my "rsyncrm" script: http://article.gmane.org/gmane.network.rsync.general/16273 -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Delay for --remove-source-files
On Tue, 2009-10-13 at 03:51 -0700, Martin Scharrer wrote: > I'm using rsync with -aP --remove-source-files to move files from one > machine to another while watching the progress. I'm under the impression > that rsync is deleting the transmitted source files on-the-fly, not at the > very end, but with a delay of 2-3 files, i.e. if 10 files are moved the > first source file is deleted after the third of fourth file got transmitted. > However, if rsync is aborted (CTRL+C) all fully source transmitted files are > deleted. > Can anyone tell me if this delay is intended behavior and if it can be > manipulated/configured differently? > My source machine has only limited space (10GB), my files are rather big (.4 > - 1.2GB each) and my bandwidth around 300-500KB, so sometimes I want to get > rid of the sent files as fast as possible without waiting for the next files > being transmitted. IIUC, the delay occurs due to the pipelining in rsync; since it is not a problem for most users, no special effort was made to avoid it. If it is a problem for you, you might consider running rsync once per file. It sounds like the files are big enough that doing so wouldn't be unreasonably wasteful. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: how can this be: "file has vanished" in --list-only?
On Thu, 2009-10-15 at 17:40 -0300, Carlos Carvalho wrote: > I'm getting "file has vanished" messages during a (recursive) > --list-only. I find it strange because I'd expect rsync to access each > file only once when just sending the receiver the file list. There's still a small gap between the "readdir" that returns a file's name and the "stat" on that file. I bet that's what you're seeing. If you want to know for sure, you can strace rsync. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Nice little performance improvement
On Thu, 2009-10-15 at 19:07 -0700, Mike Connell wrote: > Today I tried the following: > > For all subsub directories > a) Fork a "du -s subsubdirectory" on the destination > subsubdirectory > b) Run rsync on the subsubdirectory > c) repeat untill done > > Seems to have improved the time it takes by about 25-30%. It looks > like the du can > run ahead of the rsync...so that while rsync is building its file > list, the du is warming up > the file cache on the destination. Then when rsync looks to see what > it needs to do > on the destination, it can do this more efficiently. Interesting. If you're not using incremental recursion (the default in rsync >= 3.0.0), I can see that the "du" would help by forcing the destination I/O to overlap the file-list building in time. But with incremental recursion, the "du" shouldn't be necessary because rsync actually overlaps the checking of destination files with the file-list building on the source. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Nice little performance improvement
On Sat, 2009-10-17 at 12:13 -0700, Mike Connell wrote: > > Interesting. If you're not using incremental recursion (the default in > > rsync >= 3.0.0), I can see that the "du" would help by forcing the > > destination I/O to overlap the file-list building in time. But with > > incremental recursion, the "du" shouldn't be necessary because rsync > > actually overlaps the checking of destination files with the file-list > > building on the source. > > Ignoring incremental recursion for a moment. Don't ignore it, it makes a difference. > It seems to me that anything > that can warm up the file cache before it is needed would be beneficial? I didn't reason it out carefully enough; let's try again... Warming up the destination file cache decreases the amount of time the generator spends blocked on I/O. So the answer is yes, provided that the generator is the bottleneck. If incremental recursion is not used, that's almost certainly the case during the main phase of the rsync run, since the generator is checking all the destination files but the sender is only processing the small number of source files that need a transfer. But with incremental recursion, the sender and generator are checking files in parallel, so the sender may be the bottleneck depending on the relative speeds or disk configurations of the machines. (I take it that your rsync run is local. For remote runs, the network could be the bottleneck.) -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude files from backup with rsync -b
On Wed, 2009-10-21 at 17:50 +0800, Thomas Gutzler wrote: > is there a way to exclude files from being backed up when running rsync -b? No. --backup is one of many rsync options for which per-file settings might be useful but are not currently supported. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Limiting remote operations to a particular directory, and not above ?
On Wed, 2009-10-21 at 13:40 -0700, George Sanders wrote:
> I am tasked with pointing rsync transfers to valuable, live systems.
>
> The requirements include that this rsync job be run as root (rsync
> over ssh to the destination, as root) and that the --delete option be
> used.
> What would really make me feel better is if I could somehow tell
> rsync:
>
> "don't operate at all below /this/point/in/remote/filesystem" No
> matter what.
An rsync daemon is the right tool to ensure this, without a doubt.
> (I have thought of chrooting a different sshd on the remote, but I'd
> really, really, like to keep the complexity and configuration on the
> sending end and just leave these very simple remote systems alone)
If you don't want to configure the receivers in advance, your script can
invoke a single-use daemon with the configuration data passed on the
command line, like so (bash):
function quote_args {
apos="'"
bs=\\
# Escape existing single quotes.
set -- "${@//$apos/$apos$bs$apos$apos}"
# Wrap each arg in single quotes.
set -- "${@/#/$apos}"
set -- "${@/%/$apos}"
# Join the args with spaces.
echo "$*"
}
CONFIG='
[module]
path = /this/point/in/remote/filesystem
uid = root
gid = root
read only = false
'
rsync -e ssh --rsync-path="rsync --config=<(echo $(quote_args "$CONFIG"))" \
OPTIONS SRC ... rsync://HOST/module
Yes, this is pretty crazy, but it accomplishes what you want.
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: hard links on samba shares
On Fri, 2009-10-23 at 09:15 +0200, Riccardo Iaconelli wrote: > I'm encountering a bug with samba shares, rsync and hard links. I was trying > to investigate why on my "linksys" samba share i wouldn't get any hard link > transferred and rsyncs to it would take forever. To keep a long story short, > here's the output of the command, which should be pretty explanatory :-) > > mount says that: > //192.168.0.105/DISK on /samba/linksys type cifs (rw,mand) > > > [r...@popper linksys]# ls -lah test/* > -rw-rw-r-- 2 502 501 4 2009-10-23 08:53 test/file1 > -rw-rw-r-- 2 502 501 4 2009-10-23 08:53 test/file2 > > [r...@popper linksys]# rsync -avz --hard-links test/ test2/ > sending incremental file list > ./ > file1 > file2 > > sent 146 bytes received 53 bytes 398.00 bytes/sec > total size is 8 speedup is 0.04 > > [r...@popper linksys]# ls -lah test2/* > -rw-rw-r-- 1 502 501 4 2009-10-23 08:53 test2/file1 > -rw-rw-r-- 1 502 501 4 2009-10-23 08:53 test2/file2 > > Is this a bug or just samba being not reliable for hard links? :-) Probably the latter. You can always strace rsync to check that it is issuing the link(2) call and see what the result is. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: hard links on samba shares
On Fri, 2009-10-23 at 11:31 +0200, Paul Slootman wrote: > On Fri 23 Oct 2009, Matt McCutchen wrote: > > On Fri, 2009-10-23 at 09:15 +0200, Riccardo Iaconelli wrote: > > > > > > [r...@popper linksys]# rsync -avz --hard-links test/ test2/ > > > sending incremental file list > > > ./ > > > file1 > > > file2 > > > > Is this a bug or just samba being not reliable for hard links? :-) > > > > Probably the latter. You can always strace rsync to check that it is > > issuing the link(2) call and see what the result is. > > Rsync is not even trying the link(2) call, as it doesn't recognize that > the files are linked; the output would have been: > > ./ > file1 > file2 => file1 You're right. Somehow I was thinking that rsync might silently adjust its output if hard linking fails, but that doesn't make much sense and indeed that is not how rsync behaves (I tested). -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: avoid rsync deleting vanished files on receiver
On Wed, 2009-10-21 at 12:14 +0200, Hans-Christian Jehg wrote:
> I have a rather curious problem. I rsync from a fileserver shared in via
> cifs. This connection sometimes fails (don't worry, that is not the
> question I have).
> I am not at all sure where it goes wrong, but in the end the result is
> that rsync thinks certain files has disappeared. It could for instance
> go wrong in the cifs client, but also in rsync when trying to read the
> remote file (I am not using --ignore-errors) if no proper read checking
> is in place (I do not think that this is the case but I would mention it
> anyway).
> My immediate problem is that rsync proceeds happily and deletes any
> vanished files in the end (--delete-after).
> rsync stops with exit code 24, but by then the damage is already done
> (files deleted) .
On Fri, 2009-10-23 at 17:32 +0200, Hans-Christian Jehg wrote:
> The circumstance where this happens is now known, it is when requests
> to the shared in files returns "permission denied" from the file
> server...
Where are you seeing the "permission denied" text? It would help if you
would post the exact output of rsync pertaining to the affected files.
When rsync's "stat" call on a source file fails, the behavior depends on
the returned error code. If the error is ENOENT ("No such file or
directory"), rsync reports the file as vanished and sets exit code 24.
But if the error is anything else, including EACCES ("Permission
denied"), rsync sets the IOERR_GENERAL flag (which disables deletion
unless --ignore-errors is passed) and sets exit code 23. So if rsync is
truly printing "Permission denied", it makes no sense that it would then
exit with code 24.
If your source filesystem is returning ENOENT for what is really a
permission denied error, that's the filesystem's problem, not rsync's.
Disabling the deletion of vanished files would be a misfeature, as
Carlos suggested.
On Sun, 2009-10-25 at 08:52 +0100, Hans-Christian Jehg wrote:
> no the exit code is 24...
>
> I also sometimes get exit code 23, but then rsync does it correctly...
>
> What would be the effect of the file system reporting "permission
> denied" while reading directories? A list with files missing ?
Precisely; that's why rsync disables deletion in such cases.
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Optional filter files
On Tue, 2009-10-27 at 15:38 -0700, Jacob Weber wrote:
> Is it possible to call rsync and tell it to use a filter file if it
> exists, but otherwise continue without errors?
>
> If I pass "--filter=. .rsync-filter", it will fail if .rsync-filter
> doesn't exist.
>
> I know you can pass "--filter=: /.rsync-filter" to search for filter
> files in each directory. That won't fail if there aren't any such
> files. But I'm only interested in one file at the root.
No, rsync does not have such a feature. It could be added, but I would
be skeptical of letting the filter support grow organically into
something unmanageable; I'd rather see it replaced with a full scripting
language once and for all.
For now, you can test for the filter file in the script calling rsync.
Here's the syntax for bash:
filter_opt=()
if [ -e .rsync-filter ]; then
filter_opt=("--filter=. .rsync-filter")
fi
rsync ... "${filter_o...@]}" ...
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OS X 10.6 (Snow Leopard) HFS+ File Compression
On Tue, 2009-10-27 at 19:31 -0400, Tony wrote: > Are there any patches (or planned updates) to rsync v3.0.6 to handle > the HFS+ File Compression that Apple introduced with Snow Leopard? What kind of special treatment from rsync were you expecting? I read http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/3 , and as far as I can tell, the compression is handled entirely by the filesystem with no intervention from applications needed. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OS X 10.6 (Snow Leopard) HFS+ File Compression
On Tue, 2009-10-27 at 23:38 -0400, Tony wrote: > When rsync 3.0.6 copies files with HFS+ File Compression, the new > extended attribute decmpfs is not preserved, and the UF_COMPRESSED > flag is not set on the destination and the destination file is not > compressed. > > I examined the destination file as described in ars technica (with ls > and xattr from a 10.5 Leopard boot), and the compressed data is moved > from the resource fork to the data fork, and the extended attributes > '@' are removed from the file. > > As far as I know, only ditto in 10.6 can handle HFS+ File > Compression. (I even tested a 'clone' with disk utility (file copy, > not block), and it also failed (block copy, of course works). Rsync is just reading and writing files via the filesystem API; it has no access to any of the flags or xattrs used to implement the compression. I guess the filesystem doesn't compress new files by default. If it had an API to request compression, rsync could use that API when writing the destination files. Unfortunately, the API ditto is using appears to be private to Apple. See the post from brkirch beginning "The first thing that I tried to do" on this page: http://www.macosxhints.com/article.php?story=20090902223042255 So anyone interested in making rsync compress the destination files would probably have to copy the relevant code from afsctool. This could be shared as a patch; I feel quite sure it would not be adopted in the main version of rsync. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Parallel rsync's for better Performance.
On Wed, 2009-10-28 at 10:01 +0100, Matthias Schniedermeyer wrote: > On 28.10.2009 09:05, Satish Shukla wrote: > > We have huge data to sync usually everyday and I wish rsync could guarantee > > performance. > > > > I thought of spliting the directories and run parallel rsyncs on them. It > > may cost me some network, but I can control that from the MAX_RSYNC_PROCESS > > variable. Can some one evaluate pros and cons of this design?. Any help is > > heartily appreciated. > > That only works IF: > - You have SSDs (preferably good ones, both sides) > - Each rsync covers a different physical HDD (both sides) > - You have a massive Array with truck-loads of HDDs and a matching > controller or something along that line (again both sides). > - A combination of the above would also work > > Otherwise parallel rsyncs completly kill any performance you had because > normal HDDs will fall into a seek-storm, when more than 1 rsync works on > them. Asynchronous I/O may solve that, on OSes that support it. See also this RFE, on which I have just commented: https://bugzilla.samba.org/show_bug.cgi?id=5124 -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Parallel rsync's for better Performance.
On Wed, 2009-10-28 at 17:24 +0100, Matthias Schniedermeyer wrote: > On 28.10.2009 10:35, Matt McCutchen wrote: > > On Wed, 2009-10-28 at 10:01 +0100, Matthias Schniedermeyer wrote: > > > Otherwise parallel rsyncs completly kill any performance you had because > > > normal HDDs will fall into a seek-storm, when more than 1 rsync works on > > > them. > > > > Asynchronous I/O may solve that, on OSes that support it. > > No. That's a fundamental problem with ANY rotating media device. "Solve" may be an overstatement, but asynchronous I/O would at least help significantly because one process could issue many I/O requests to the same area of disk at once, and the disk scheduler could fulfill all of those requests before seeking elsewhere. Without asynchronous I/O, after the scheduler fulfills one request, it is left to either seek or wait for the process to issue another request. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Operation not supported (95)
On Wed, 2009-10-28 at 18:55 +, Kay wrote:
> I'm running into a problem that I'm sure which part of the process
> fails. I use wget (OSX Leopard, wget 3.0.6 compiled from source) to
> backup my ~/Pictures directory to a linux server, also with wget 3.0.6
> (Debian). In persuit of being sure I get everything from HFS filesystem
> (resource fork/metadata), I'm getting the following error:
>
> rsync: rsync_xal_set:
> lsetxattr("Pictures/...[snip].../IMG_7766/IMG_7766.dng","user.com.apple.FinderInfo")
>
> failed: Operation not supported (95)
>
> I have search through the archives and only found this error in what
> seemed to be an error relating to selinux (which I'm not using). The
> target filesystem is ext3 with acl enabled.
Does the target filesystem have user extended attributes enabled with
the "user_xattr" mount option? If not, rsync would get "Operation not
supported" when it tries to set them.
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Parallel rsync's for better Performance.
On Wed, 2009-10-28 at 23:46 +0100, Matthias Schniedermeyer wrote: > On 28.10.2009 18:27, Matt McCutchen wrote: > > On Wed, 2009-10-28 at 17:24 +0100, Matthias Schniedermeyer wrote: > > > On 28.10.2009 10:35, Matt McCutchen wrote: > > > > On Wed, 2009-10-28 at 10:01 +0100, Matthias Schniedermeyer wrote: > > > > > Otherwise parallel rsyncs completly kill any performance you had > > > > > because > > > > > normal HDDs will fall into a seek-storm, when more than 1 rsync works > > > > > on > > > > > them. > > > > > > > > Asynchronous I/O may solve that, on OSes that support it. > > > > > > No. That's a fundamental problem with ANY rotating media device. > > > > "Solve" may be an overstatement, but asynchronous I/O would at least > > help significantly because one process could issue many I/O requests to > > the same area of disk at once, and the disk scheduler could fulfill all > > of those requests before seeking elsewhere. Without asynchronous I/O, > > after the scheduler fulfills one request, it is left to either seek or > > wait for the process to issue another request. > > And "same disc region" is kind of a problem. In most modern filesystems > inodes can be pretty random so you can't for sure sort the files by > inode, or something like that. I wasn't implying any effort on the process's part to choose files in the same disk region. Your statement that running parallel rsyncs creates a worse seek storm than one rsync alone is based on the assumption that each rsync tends to process files that are together on disk, and I was simply referring to that assumption. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync ignoring files beginning with . ??
On Thu, 2009-10-29 at 10:11 -0400, Michael W. Cocke wrote: > Quick question (I hope) I'm trying to use rsync 3.0.4 and opensuse 11.1 > to sync an entire /home tree. How to I get rsync to not skip files > beginning with a . (period) ? Rsync does not treat any filenames specially. There must be something else going on. What is the full rsync command you are using? What is an example of one file that is being skipped? -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Operation not supported (95)
On Thu, 2009-10-29 at 21:11 +, List Reader wrote:
> While I haven't
> found the exact cause of the error, in my search I did find some
> suggestions that XFS is a better choice for backing up from HFS+, so I
> tried that and this error disappeared.
>
> This one remains though:
>
> rsync: rsync_xal_set: lsetxattr("Pictures/...[snip].../IMG_0960
> (2)/IMG_0960.tiff","user.com.apple.ResourceFork") failed: Argument list
> too long (7)
> rsync: rsync_xal_set:
> lsetxattr("Pictures/...[snip].../Image0001.TIF","user.com.apple.ResourceFork")
>
> failed: Argument list too long (7)
It looks like you've hit a limit on the size of the attribute value. I
tested on both xfs and reiserfs and saw a limit at exactly 65536 bytes;
I don't know whether this is a coincidence or the limit is actually in
Linux.
I don't see an easy way around this, so your options would be:
1. Figure out whether you actually need the massive resource-fork data:
make a copy of the source file, run "setfattr -x com.apple.ResourceFork"
on the copy, and see if you've lost anything important. If not, just
keep the stripped file.
2. If you do need the resource fork, you could have your backup script
split the file into a plain data file and a separate file with the xattr
information, using the "xattr" command or something that will make a
standard AppleDouble file if you like that. Put the plain data file in
a separate tree and add that tree as the first source argument to rsync,
so that the plain data file overlays the original and rsync won't get
xattr errors. Reply if you want to take this approach and need more
help.
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Unix file ownership for wide open directories
On Fri, 2009-10-30 at 12:16 -0400, JESSE CARROLL wrote: > Forgive me if the answer is obvious but I've googled and searched the > archives but I can't seem to find a good solution. > > Scenario on a Solaris system: > > ls -ld /foo > drwxrwxrwx 2 user1other512 Oct 30 16:05 /foo > > ls -l /foo/* > total 0 > -rw-r--r-- 1 user1other 10 Oct 30 16:05 /foo/file_a > -rw-r--r-- 1 user2staff 30 Oct 30 16:05 /foo/file_b > > (Yes I know wide open directories are evil, but the application folks > do strange things.) > > If I use rsync as user1 all the files on the destination server are > owned by user1. I.E. > > ls -l /foo/* > total 0 > -rw-r--r-- 1 user1other 10 Oct 30 16:05 /foo/file_a > -rw-r--r-- 1 user1other 30 Oct 30 16:05 /foo/file_b > Any suggestions on preserving the file ownership in this case? The -o (--owner) option? -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Unix file ownership for wide open directories
On Fri, 2009-10-30 at 12:16 -0400, JESSE CARROLL wrote: > Forgive me if the answer is obvious but I've googled and searched the > archives but I can't seem to find a good solution. > > Scenario on a Solaris system: > > ls -ld /foo > drwxrwxrwx 2 user1other512 Oct 30 16:05 /foo > > ls -l /foo/* > total 0 > -rw-r--r-- 1 user1other 10 Oct 30 16:05 /foo/file_a > -rw-r--r-- 1 user2staff 30 Oct 30 16:05 /foo/file_b > > (Yes I know wide open directories are evil, but the application folks > do strange things.) > > If I use rsync as user1 all the files on the destination server are > owned by user1. I.E. > > ls -l /foo/* > total 0 > -rw-r--r-- 1 user1other 10 Oct 30 16:05 /foo/file_a > -rw-r--r-- 1 user1other 30 Oct 30 16:05 /foo/file_b > > I'm trying to avoid using rsynd as we're paranoid about security. (We > change root password on a regular basis. The solutions indicate that I > may have to change the secrets file each time.) > > Any suggestions on preserving the file ownership in this case? Whoops, it would help if I read your whole message before replying. In order to have the power to set the destination ownership to match the source ownership, the receiving rsync must run as root; this requirement is imposed by the OS. You'll need to use ssh as root or an rsync daemon running as root. If you want to piggyback on the existing well-maintained root password but like some of the extra protections of an rsync daemon, you can use an ssh-invoked single-use daemon, as described in the "USING RSYNC-DAEMON FEATURES VIA A REMOTE-SHELL CONNECTION" section of the rsync man page. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Almost desperate on rsync from macbook to NAS
On Sat, 2009-10-31 at 17:06 +0100, Boniforti Flavio wrote: > I'm almost getting exhausted, thus I have to bother you people to get some > help. That's a non sequitur. Check http://www.catb.org/~esr/faqs/smart-questions.html#id382403 . > I simply want to sync my "iTunes Music" folder from my MacBook over to my NAS. > The scenario is like this: > > source path: "/Users/bonny/Music/iTunes/iTunes Music/" (in here there > are all the subfolders I want to sync over to the NAS); > destination: "rsync://r...@192.168.1.100/musica/iTunes Music/" > > What I did is to copy the stuff over "by hand" (tens of GB!!!)... now > I sometimes add new music to my iTunes Library, therefore I don't want > to sync all stuff, just add to the NAS what's been added on my MacBook > and delete on the NAS what has gone from my MacBook. > > I tried many many ways, but without any success!! What is the nature of the failure? Files ending up in the wrong place, or rsync not seeing the old files and wanting to retransfer everything, or not connecting to the rsync daemon at all, or what?? > I tried to play around with trailing slashes, but I got tired while > not succeeding :-/ Putting a trailing slash on the source should have worked. (Whether there is one on the destination shouldn't matter.) -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: read errors mapping "/path/file": No data available (in version 3.0.6)
On Mon, 2009-11-02 at 20:33 -0200, Carlos Carvalho wrote: > Got this in the log: > > WARNING: path/file failed verification -- update discarded (will try again). > rsync: read errors mapping "/path/file": No data available (61) > > The first line is rare but understandable. It's the second one that I > find strange. What does "no data available" mean? The file exists on > the origin (and on the destination). The second attempt should have > worked normally. This happened with --link-dest. It looks like rsync generates that errno internally if a source file is truncated while the sender is reading it: http://gitweb.samba.org/?p=rsync.git;a=blob;f=fileio.c;h=0faa619540e6892e16b1fe13ee55a5d3bd1c6fb6;hb=HEAD#l249 -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exit status 13 in version 3.1
On Thu, 2009-10-29 at 13:20 -0200, Carlos Carvalho wrote: > Got this in the log: > > rsync error: errors with program diagnostics (code 13) at log.c(340) > [generator= > 3.1.0dev] I got this too on a big local run backing up my system to an external disk using rsnapshot. /etc/rsnapshot-rsync -aAXxx --del --numeric-ids --relative \ --delete-excluded --remote-option=--munge-links --filter=P_/home/ \ --filter=P_/boot/ --exclude=/home/*/.gvfs \ --link-dest=/media/crypt-backup/snapshots-mattlaptop2/occasional.0/ml2/ \ /home /media/crypt-backup/snapshots-mattlaptop2/.sync/ml2/ rsync error: errors with program diagnostics (code 13) at log.c(340) [sender=3.1.0dev] rsync error: received SIGUSR1 (code 19) at log.c(340) [generator=3.1.0dev] (The /etc/rsnapshot-rsync wrapper script is just prepending another --link-dest option to the command line.) This probably has to do with Wayne's recent I/O changes. When I have some time tonight, I'll see if I can reproduce the problem and capture some more information that might help Wayne figure it out. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exit status 13 in version 3.1
On Tue, 2009-11-03 at 12:25 -0500, Matt McCutchen wrote: > On Thu, 2009-10-29 at 13:20 -0200, Carlos Carvalho wrote: > > Got this in the log: > > > > rsync error: errors with program diagnostics (code 13) at log.c(340) > > [generator= > > 3.1.0dev] > > I got this too on a big local run backing up my system to an external > disk using rsnapshot. It looks like I was just out of disk space, but the "No space left on device" message was not making it through the pipeline. I reproduced this with a single file and a small reiserfs image: $ rsync-dev big-file small-fs/ rsync error: errors with program diagnostics (code 13) at log.c(340) [sender=3.1.0dev] $ rsync-dev --msgs2stderr big-file small-fs/ rsync: write failed on "/PATH-TO/small-fs/big-file": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(334) [receiver=3.1.0dev] rsync error: errors with program diagnostics (code 13) at log.c(340) [sender=3.1.0dev] rsync error: received SIGUSR1 (code 19) at main.c(1387) [generator=3.1.0dev] Since I'm doing a local run, I can probably use --msgs2stderr for now. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: error in logging (version 3.1.0)
On Sat, 2009-11-07 at 00:36 -0200, Carlos Carvalho wrote: > I use rsync -ii so that it logs *all* actions. Further, I don't use > any delete option but use --force. This means that if there's a name > collision rsync deletes files or directories. Here's an example: > > 0 0*deleting rwxrwsr-x 1969/12/31-21:00:00 > scm/linux/kernel/git/amit/vs-kernel.git/objects/38 > 0 0*deleting rwxrwsr-x 1969/12/31-21:00:00 > scm/linux/kernel/git/amit/vs-kernel.git/objects I guess some ancestor of "scm/linux/kernel/git/amit/vs-kernel.git/objects" became a non-directory? That seems unlikely to me. > Therefore I didn't expect to find any of these directories after the > sync is finished. However I get: > > % ls scm/linux/kernel/git/amit/vs-kernel.git/objects > 09 15 2d 82 a8 bc cf e0 ff > > It might be that for some reason rsync created the directory again. I don't think so. I only know of one rare case in which rsync would delete and recreate a directory, and I don't think it applies here: when the file list contains an ancestor and a descendant but not the directory itself, which can only happen with --relative --no-implied-dirs and protocol <= 29. > BUT *no other lines* with > scm/linux/kernel/git/amit/vs-kernel.git/objects appear in the log. So > we have two possibilities: either rsync lied and didn't delete the > directory, or it created it again and didn't tell it in the log. Do you have a reproducible test case? -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Input/Output error 5 ?
On Thu, 2009-11-05 at 09:20 -0500, Michael W. Cocke wrote: > What, exactly, does "input/output error 5" mean? (Yes I googled it, but > all I can find is a lot of people getting the error, not what it means). Typically, that there is something wrong with the filesystem or the disk, which is out of rsync's control. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exit status 13 in version 3.1
On Sat, 2009-11-07 at 09:38 -0800, Wayne Davison wrote: > On Wed, Nov 4, 2009 at 8:05 PM, Matt McCutchen > wrote: > > With commit 84c11e85a4c4a12ecacba24afe9617222e4361e6, I get > different > > output, but still not the desired "No space left on device": > > > Yeah, that's the long-standing issue where a fatal error on the server > side can cause the client side to get a socket error trying to write > to the socket before it has a chance to read the error(s) from the > socket. The latest git archive finally has a fix for this. I tested commit 2907af472d1f33b3c422cb9f601c121b242aa9c7 and, again, the output is different but the problem is not fixed: $ rsync-dev big-file small-fs/ rsync: connection unexpectedly closed (146 bytes received so far) [sender] $ rsync-dev --msgs2stderr big-file small-fs/ rsync: write failed on "/PATH/TO/small-fs/big-file": No space left on device (28) rsync: connection unexpectedly closed (36 bytes received so far) [sender] rsync error: error in file IO (code 11) at io.c(181) [receiver=3.1.0dev] -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Take2: rsync over ssh with --link-dest
On Sun, 2009-11-08 at 21:01 -0500, Alex wrote: > > Overriding the server command line is generally a bad idea for the > > reasons you went on to note. Instead, use an rsync daemon. You can > > even have the daemon invoked over ssh with a forced command, see: > > > > https://bugzilla.samba.org/show_bug.cgi?id=4163#c4 > > That looks like the way to go. I've got it set up and running for a > few days now, and it's working okay. Good. > I'm a bit disappointed that I > can't do it by overriding the server command line Why? Does it have some advantage over the use of an rsync daemon (I can't imagine what)? One can't really expect overriding the server arguments not to lead to the inconsistent behavior you saw. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Take2: rsync over ssh with --link-dest
On Mon, 2009-11-09 at 00:28 -0500, Alex wrote: > >> I'm a bit disappointed that I > >> can't do it by overriding the server command line > > > > Why? Does it have some advantage over the use of an rsync daemon (I > > can't imagine what)? One can't really expect overriding the server > > arguments not to lead to the inconsistent behavior you saw. > > Simplicity. I'd like to be able to have the server side feed me all > data beginning with the path I specify, such /backup/serverA, In other words, you were specifying a bogus source path and counting on sshd to override it with the real one. Clever. I didn't catch that the first time. Realize that the ability to set a module's "path" in the rsyncd.conf file and have the client refer to the module by a well-known name gives you the same functionality, and actually more, because the client can specify a subdirectory path within the module. > and have > the client side be able to "--include=/usr/bin" to be able to > 'download' only the data from /backup/serverA/usr/bin and exclude > everything else. That --include means "exclude everything else" is a common misconception; see the man page. But I imagine the issue will become moot if you have the client specify a subdirectory of the module instead of using filters. > Instead, I have about eight sections of my rsyncd.conf to do this; one > for each directory tree that I want to back up to a different > location, primarily based on dates, to have multiple incrementals. If the daemon solution is looking fundamentally more complicated than the set of forced server commands you previously had, you're probably going about it the wrong way and I could help if you post the rsyncd.conf file. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: max file size
On Mon, 2009-11-09 at 11:43 +0100, Heinz-Josef Claes wrote: > does anybody know what's the maximum file size (terabytes?) when using rsync > with options --checksum and / or --inplace? > > What file sizes have been tested in reality? Are there any experiences using > rsync (with --checksum and / or --inplace) for big files with several / > dozens > or terabytes? I don't believe rsync has a fixed maximum size other than "what can fit in 64 bits", but I can't speak to any reliability issues that might come up with extremely large files. For what purpose are you considering --checksum? In the case where the file's size hasn't changed (probably true for large image files), it will add an extra full read of the file on both sides before the transfer begins, which would be very expensive for multi-terabyte files. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Yet another include/exclude question
On Tue, 2009-11-10 at 09:45 +0800, Thomas Gutzler wrote: > Thanks everyone for your help, I've got what I want. > > Wayne Davison wrote: > > On Sun, Nov 8, 2009 at 10:21 PM, Thomas Gutzler > > mailto:thomas.gutz...@gmail.com>> wrote: > > > > I thought --include="/this_dir/" --include="/this_dir/***" would do it, > > but it doesn't. The exclude * seems to overwrite the include matches: > > [sender] hiding file this_dir/foo because of pattern * > > > > > > Order is important. Whatever matches first, is what takes effect. > > Also, *** matches both the dir and its contents, so you could use: > > > > --include='/this_dir/***' --exclude=* > > It seems to be very picky about the order. Thanks for pointing that out. > My first attempt has been > --include="*/" --include="*.foo" --include="*.bar" > --include="/this_dir/***" --exclude="*" > which did nothing than *.foo and *.bar. Shuffling it around, I found that > --include="*/" --include="/this_dir/***" --include="*.foo" > --include="*.bar" --exclude="*" > does what I want and it even makes sense. Those two commands should be equivalent. If you have a reproducible case in which they aren't, please share it and we can see if there's a bug. I tried the first command and it worked fine for me. That is, after I fixed a typo I made in the name of "this_dir", which left the --include="/this_dir/***" nonfunctional and gave a result like the one you cited above: [sender] hiding file this-dir/one because of pattern * -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --fuzzy search over to-be-deleted files to catch moved files and directories
On Tue, 2009-11-10 at 17:55 +0100, H. Langos wrote: > Now the question is, if fuzzy search could be extended to search for > moved files across directory borders. If for example I move pictures > aound to reorganize them in a different directory hierarchy, or if I > move whole directories around as I reorganize part of my music files. Consider the --detect-renamed option provided by the maintained patch "detect-renamed.diff". It will find moved files that match exactly according to the "quick check" in effect (size + mtime or checksum). It doesn't calculate name similarity like --fuzzy because that would be prohibitively expensive in the current implementation. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Take2: rsync over ssh with --link-dest
On Tue, 2009-11-10 at 22:47 -0500, Alex wrote:
> > If the daemon solution is looking fundamentally more complicated than
> > the set of forced server commands you previously had, you're probably
> > going about it the wrong way and I could help if you post the
> > rsyncd.conf file.
>
> Okay, great, it's included below.
> use chroot = no
> syslog facility = local5
>
> [cdrive]
>list = true
>path = /backup/servers/xxx.xxx.228/cdrive
>comment = 228-cdrive
>read only = true
>gid = root
>uid = root
[more modules like this...]
> The client side command looks something like this:
>
> rsync -avz --delete --numeric-ids --exclude-from=/backup/exclude-file.txt \
> --link-dest=/backup/seg/serverA/xxx.xxx.228/cdrive/20091109 \
> rsync://serverA/cdrive/20091110/ \
> /backup/seg/serverA/xxx.xxx.228/cdrive/20091110
Your daemon setup looks exactly right. I don't understand why you
consider it more complicated than the previous setup. Previously you
had one SSH authorized key for each backup point with the appropriate
path forced; now the daemon configuration contains the same information.
The daemon configuration looks more verbose, but that could be helped by
moving the parameters that are the same for all modules ("list", ...) to
the global section at the top, or you could even generate the
configuration file with M4. The client command is the same except for
the specification of a module name instead of an SSH key. Am I missing
something?
> > Realize that the ability to set a module's "path" in the rsyncd.conf
> > file and have the client refer to the module by a well-known name gives
> > you the same functionality, and actually more, because the client can
> > specify a subdirectory path within the module.
>
> I'm not sure I understand fully, but this may be what I'm already
> doing. An example would be helpful.
You are indeed doing this. Your source is:
rsync://serverA/cdrive/20091110/
The rsync://serverA/cdrive/ is mapped to
/backup/servers/xxx.xxx.228/cdrive according to the configuration file,
resulting in:
/backup/servers/xxx.xxx.228/cdrive/20091110/
In contrast, the forced-server-command approach would clobber the
20091110/ part, preventing you from setting up the --link-dest properly.
Is it clear now?
> To make it even more complicated, on the receiving (client) side, I
> have to pass the backup script I use through sed to rewrite the date
> which gets appended to the source path, to update every day so I can
> use the --link-dest rsync argument to create a backup of only the
> changes from day-to-day, and hard link the rest from the reference
> location, which is usually the previous day.
It seems to me that you would have had to do the same thing under the
forced-server-command setup. No?
In any case, sed is overkill to vary the date. You can just use a shell
variable:
DATE=$(date +%Y%m%d)
rsync ... /backup/seg/serverA/xxx.xxx.228/cdrive/$DATE
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: does it make sense to run rsync over ftp (curlftpfs)?
On Tue, 2009-11-10 at 08:40 +0100, Mac User FR wrote: > In another hand, if bandwidth is not your problem, with this method > rsync should (to be confirmed by an expert, what I am not) still use > incremental updates on your files if your ftp volume support append or > other file modification function that rsync needs. With --append, rsync will append to the destination file. With --no-whole-file --inplace, it will write only the changed blocks, but it will read the whole file first, which may defeat the purpose. Rsync is at the mercy of curlftpfs as well as the FTP server to perform any partial updates efficiently; I don't care to research their capabilities myself. The main reason to use rsync over an impoverished virtual filesystem would be to take advantage of rsync's non-performance-related features, such as filters. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Translating user names during transfer.
[Please start a new thread for a new topic.] On Wed, 2009-11-11 at 12:38 -0500, Dave Mielke wrote: > Is there a way to translate file ownership during transfer. For example, can > it > be done such that all files owned by user1 end up being owned by user2, and > all > files owned by user3 end up being owned by user4. The actual problem we're > trying to solve is that some, but not all, users don't exist on the target > system so we'd like the ownership for users which do exist to remain but to > have a default owner (or, even better, set of owners) for the rest. The --usermap option, which is integrated in the current development rsync and available in a maintained patch "usermap.diff" for rsync 3.0.*, provides some flexibility in mapping file owners. Unfortunately, looking at the man page, there doesn't appear to be a way to map all names that are missing on the receiving side. It's a very natural extension, so I think it ought to be added, but whether anyone is interested in actually doing it, we'll see. Note that there's no such thing as a set of owners on POSIX filesystems. Maybe you would want an ACL granting multiple people access, but that's something you would have to do yourself with a "find ... -exec setfacl ..." command after running rsync. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Does files-from work with --delete?
On Wed, 2009-11-11 at 11:55 -0800, Philip Pokorny wrote: > If modifying the file sending code to detect the link_stat "not found" > error and if --delete is enabled, sending that as a delete request (is > there such a thing in the protocol?) is the right thing, then I'll > work > on a code patch. That feature exists in the current development rsync, under the name --delete-missing-args. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: anchoring of patterns for per-directory merge files
On Wed, 2009-11-11 at 17:14 +, Andy Smith wrote: > I'm extremely confused about anchoring of patterns for per-directory > merge files, as what I see seems to not be the behaviour the > documentation suggests. > > On the receiver side I have rsync version 3.0.3 protocol version > 30. On the sender I have rsync version 2.6.9 protocol version 29. > I'm calling: > > rsync --filter='dir-merge /.rsync-filter' -avz --delete \ > --numeric-ids --relative --delete-excluded --sparse --hard-links \ > --rsh='/usr/bin/ssh -4 -i /root/.ssh/rsync -q' \ > r...@corona:/data \ > /data/backup/.sync/corona/ > > On the sender, I have in the file "/data/home/andy/.rsync-filter": > > - /tmp/ > > The documentation leads me to believe that this should exclude a > directory called "tmp/" located only in the same directory as the > merge file. What happens is, nothing gets excluded. This case works just fine for me with rsync 3.0.6 on both sides. Please upgrade. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exit status 13 in version 3.1
On Sun, 2009-11-08 at 01:57 -0500, Matt McCutchen wrote: > I tested commit 2907af472d1f33b3c422cb9f601c121b242aa9c7 and, again, the > output is different but the problem is not fixed: > > $ rsync-dev big-file small-fs/ > rsync: connection unexpectedly closed (146 bytes received so far) [sender] > > $ rsync-dev --msgs2stderr big-file small-fs/ > rsync: write failed on "/PATH/TO/small-fs/big-file": No space left on device > (28) > rsync: connection unexpectedly closed (36 bytes received so far) [sender] > rsync error: error in file IO (code 11) at io.c(181) [receiver=3.1.0dev] I tested commit cece2e3f5e335b8d1bd0862dbc9edbf2d5a4f5dd and the problem is finally fixed: $ rsync-dev big-file small-fs/ rsync: write failed on "/PATH/TO/small-fs/big-file": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(334) [receiver=3.1.0dev] The output is not as clean as Wayne may have intended for remote pushes and pulls, but at least the critical information is still there: $ rsync-dev localhost:PATH/TO/big-file small-fs/ rsync: write failed on "/PATH/TO/small-fs/big-file": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(334) [receiver=3.1.0dev] rsync: connection unexpectedly closed (35 bytes received so far) [generator] $ rsync-dev big-file localhost:PATH/TO/small-fs/ rsync: write failed on "/PATH/TO/small-fs/big-file": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(334) [receiver=3.1.0dev] rsync: [sender] write error: Broken pipe (32) -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exit status 13 in version 3.1
On Sat, 2009-11-07 at 09:38 -0800, Wayne Davison wrote: > Yeah, that's the long-standing issue where a fatal error on the server > side can cause the client side to get a socket error trying to write > to the socket before it has a chance to read the error(s) from the > socket. The latest git archive finally has a fix for this. It looks like the implementation has the receiver hang around for a hard-coded 10 seconds, accepting data from the sender and discarding it. That's a hack: I don't like to have the sender dependent upon this special cooperation from the receiver in the event of abnormal termination. It seems to me that when the sender hits a write error, it could just read messages on a best-effort basis before exiting, as in the old IO code. Is this approach unworkable for some reason? -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Does files-from work with --delete?
> That feature exists in the current development rsync, under the name > --delete-missing-args. > > = > OK, so I got the latest HEAD snapshot and it compiles, but it doesn't > pass all the built-in tests. Currently, the tests pass for me. We would appreciate your help in figuring out why they aren't passing for you so the problem can be corrected, but that's a separate issue. > > How confident are you in the current state of 3.1.0.pre and the > nightly snapshots? Should I be concerned about running this on > production data? Not confident. The best approach for now is probably to backport the --delete-missing-args changes to 3.0.6. I've done so for you, here: https://mattmccutchen.net/rsync/rsync.git/?a=shortlog;h=refs/heads/hacks/missing-args-v3.0.6 -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --fuzzy search over to-be-deleted files to catch moved files and directories
Attempting to address each of your questions, here and then in your other message... On Wed, 2009-11-11 at 12:17 +0100, H. Langos wrote: > > It will find moved files that match exactly > > according to the "quick check" in effect (size + mtime or checksum). > > That is basename+size+mtime or basename+checksum, right? No, a basename match is not a requirement (hence the ability to detect renames), but it is a tie-breaker. > How does "--detect-renamed" interact with "--fuzzy" and "--delete-after"? --detect-renamed and --fuzzy are two different means of finding basis files that overlap in some cases but do not really interact. --detect-renamed considers the whole destination using the quick check, while --fuzzy considers only the same destination subdir using size+mtime or otherwise name similarity. --delete-before and --delete-during may reduce the effectiveness of --fuzzy, as stated in the man page description of --fuzzy, but they do not affect --detect-renamed since --detect-renamed actually works during the delete pass. > > It doesn't calculate name similarity like --fuzzy because that would > > be prohibitively expensive in the current implementation. > Only files of the same size should be > candidates to start with, right. No, the name similarity calculation I'm talking about is the fallback to select a similar basis file when no available destination file passes the quick check, so it does not require a size match. > Why would it be so expensive? Wayne said so here: https://bugzilla.samba.org/show_bug.cgi?id=3392#c11 I haven't done any tests it myself. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --fuzzy search over to-be-deleted files to catch moved files and directories
On Wed, 2009-11-11 at 14:32 +0100, H. Langos wrote: > Unfortunately it still does not solve my problem. "--detect-renamed" > hardlinks when it moves deleted files to its temp directory. But then > it seems to copy the file to its new location. > > This breaks the space saving property of rsnapshot (hardlinking files that > didn't change since the last snapshot). > > So in order to realy work in my setup one would have to check if the delta > transfer did change any data at all. If it didn't a hardlink would be made > instead of a copy. Ah. The RFE for the check you describe is: https://bugzilla.samba.org/show_bug.cgi?id=5583 A currently available alternative is the --detect-moved option added by "detect-renamed-lax.diff", which will accept a file matching in basename + quick check directly, without running the delta-transfer algorithm. This is risky, but not quite as risky as --detect-renamed-lax. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Does files-from work with --delete?
On Thu, 2009-11-12 at 20:11 -0800, Philip Pokorny wrote: > === I uploaded the testtmp directory contents with the logs and > results to: > http://www.mindspring.com/~ppokorny/rsync-3.1.0dev-testtmp.tar.gz Ah. You ran the testsuite as root, and the daemon tests are not set up to cope with that. The testsuite should pass if you run it as an unprivileged user. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Daemon tests broken when running as root
On Thu, 2009-11-12 at 23:56 -0500, Matt McCutchen wrote: > Ah. You ran the testsuite as root, and the daemon tests are not set up > to cope with that. The testsuite should pass if you run it as an > unprivileged user. The breakage was introduced in commit 7f367bb1b4348ba1edb863ee13f8281471785e09 , which added support for supplementary groups. It also made the daemon attempt to honor "uid" and "gid" parameters even if not root and abort the run upon failure. I'm guessing the "uid = 0" and "gid = 0" in the test daemon configuration were disabled so that the daemon tests could run for unprivileged users. But that breaks running as root, because the daemon drops privileges to "nobody". The proper fix is to emit the "uid" and "gid" parameters only when running as root. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --detect-renamed --detect-moved and -b
On Wed, 2009-11-11 at 20:27 +0800, Thomas Gutzler wrote: > I played around with the combination of rsync --detect-renamed > --detect-moved and -b. You can't combine the --detect-renamed and --detect-moved options because they make use of the partial dir in incompatible ways. The last option on the command line takes priority. (Yes, the implementation of --detect-moved is a hack. No, I don't care to improve it myself.) > Given the following tree: > src/dir/file > dest/src/dir/file > I renamed src/dir to src/dir2 and ran the following command: > rsync -a --detect-renamed --detect-moved --delete -b --backup-dir=bak src dest > ending up with: > dest/bak/src/dir/file > > Is it supposed to do that? I was expecting it to detect the renamed > directory (or moved file) and exclude it from the backup. Yes, the current design is that a deleted destination file is backed up even if it was also detected as a rename. In addition, your case tickles an awkward-to-fix bug in the --detect-renamed implementation: renames into new directories are not processed because their partial dirs are not available yet. See: http://lists.samba.org/archive/rsync/2007-December/019469.html -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exit status 13 in version 3.1
On Thu, 2009-11-12 at 21:47 -0800, Wayne Davison wrote: > On Thu, Nov 12, 2009 at 5:47 PM, Matt McCutchen > wrote: > It looks like the implementation has the receiver hang around > for a > hard-coded 10 seconds, accepting data from the sender and > discarding it. > > No, it sets a timeout of 10 seconds (i.e. 10 seconds of inactivity), You're right, my mistake. > which in the new protocol should never be reached because the "we're > exiting with an error" message gets everyone to die in unison. Unless the network is slow. IMO, hard-coding values like this should be avoided when an easy alternative exists. > The necessity of discarding data is there due to the pipelining nature > of rsync, particularly if the error is coming from the receiver. I understand that the data discarding serves to avoid giving the sender a write error so that it survives to read the message explaining the error exit. My point is that it's a clunky way to achieve the goal, and it would be simpler for the sender to just keep reading after a write error. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: max file size
On Mon, 2009-11-09 at 18:20 +0100, Heinz-Josef Claes wrote: > Am Montag, 9. November 2009 17:48:35 schrieb Matt McCutchen: > > On Mon, 2009-11-09 at 11:43 +0100, Heinz-Josef Claes wrote: > > > does anybody know what's the maximum file size (terabytes?) when using > > > rsync with options --checksum and / or --inplace? > > > > > > What file sizes have been tested in reality? Are there any experiences > > > using rsync (with --checksum and / or --inplace) for big files with > > > several / dozens or terabytes? > > > > I don't believe rsync has a fixed maximum size other than "what can fit > > in 64 bits", but I can't speak to any reliability issues that might come > > up with extremely large files. > > > I've read about a fix for overrun checksum buffers with more than some > hundred > terabytes but that was just something undefined . . . Indeed, I forgot about that. The delta-transfer algorithm doesn't work for files longer than 2^31 blocks. With the default maximum block size of 2^17, the limit is 2^48 bytes or 256 TB. You could stretch the limit by fixing a larger block size with --block-size . See: https://bugzilla.samba.org/show_bug.cgi?id=5459#c2 > > For what purpose are you considering --checksum? In the case where the > > file's size hasn't changed (probably true for large image files), it > > will add an extra full read of the file on both sides before the > > transfer begins, which would be very expensive for multi-terabyte files. > > I want to check if the following is possible: > > 1. transport a big block of data (several terabytes) physically from location > A to location B (very long distance) via tapes (or disks). > (Location A and B use different storage technologies.) > > When the tapes arrive in location B, the block of data has changed in > location > A (a program / OS is running and storing data in it). > > 2. shutdown application / OS in location A, rsync the delta between Location > A > and B online, then restart the system in location B. > > (Perhaps step 2 has to be done multiple times.) Since the source and destination versions are practically certain to differ, --checksum would serve no purpose. See the man page description of --checksum. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --detect-renamed --detect-moved and -b
On Fri, 2009-11-13 at 14:40 +0800, Thomas Gutzler wrote: > Matt McCutchen wrote: > > You can't combine the --detect-renamed and --detect-moved options > > because they make use of the partial dir in incompatible ways. The last > > option on the command line takes priority. (Yes, the implementation of > > --detect-moved is a hack. No, I don't care to improve it myself.) > > I see. That's ok though, because I'm much more interested in the case > that a user renames a directory which contained 100G of data than the > case that a user renames 100G worth of files. You may be misunderstanding the options. --detect-renamed can work across directories, i.e., it detects renames in the sense of rename(2). --detect-renamed-lax is a variation that accepts a destination file that passes the quick check directly instead of just using it as a basis (risky), and --detect-moved is a further variation that requires a basename match (less risky). (Wayne, I told you --detect-moved should have been named --detect-moved-lax!) > > In addition, your case tickles an awkward-to-fix bug in the > > --detect-renamed implementation: renames into new directories are not > > processed because their partial dirs are not available yet. See: > > > > http://lists.samba.org/archive/rsync/2007-December/019469.html > > Great. I'm running an --include='*/' --exclude='*' anyway, so I've got > the directories. The point is, the new destination directory has to exist when the old destination file is considered for a rename. Rsync might not have created the new directory yet if it is later in the sort order or if --detect-renamed is piggybacking on --delete-before. > > Yes, the current design is that a deleted destination file is backed up > > even if it was also detected as a rename. > Any chance for a --exclude-moved-from-backup flag? I'm not going to pursue it myself. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Does files-from work with --delete?
On Fri, 2009-11-13 at 08:43 -0800, Wayne Davison wrote: > On Thu, Nov 12, 2009 at 6:07 PM, Matt McCutchen > wrote: > The best approach for now is probably to backport the > --delete-missing-args changes to 3.0.6. > > In the future, I'd suggest starting with the head of the b3.0.x > branch. That currently gets you one extra commit, an xattr-related > memory fix (2daed024b17a2cafb956e12581c25119d07a5950). Sure, provided that you keep b3.0.x stable all the time rather than just at releases. I've redone the missing-args backport: https://mattmccutchen.net/rsync/rsync.git/?a=shortlog;h=hacks/missing-args-b3.0.x -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Stability of rsync 3.1.0dev (Re: Does files-from work with --delete?)
On Fri, 2009-11-13 at 08:38 -0800, Wayne Davison wrote: > On Thu, Nov 12, 2009 at 3:56 PM, Philip Pokorny > wrote: > How confident are you in the current state of 3.1.0.pre and > the nightly snapshots? Should I be concerned about running > this on production data? > > Personally, I'm almost ready to start using it in production. The > 3.1.0dev code prior to the I/O changes was in production-ready shape, > and the I/O overhaul has been testing well so far (I use it for all > the personal rsyncing needs). Any failure cases from the I/O code > should result in the stopping of the transfer, not some kind of > corruption, and I haven't found anything in quite a while. 3.1.0dev needs to win back my trust after a bad experience trying to do a backup a few weeks ago. First I got an out-of-space error that rsync didn't report: http://lists.samba.org/archive/rsync/2009-November/024135.html And once I freed up disk space, I got an "unexpected tag 50" (I think), which I held off on reporting pending a reproducible case. I haven't had a chance to try another backup with 3.1.0dev yet. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: max file size
On Fri, 2009-11-13 at 12:36 +0100, Heinz-Josef Claes wrote: > On Fri, 13 Nov 2009 01:38:48 -0500 > Matt McCutchen wrote: > > On Mon, 2009-11-09 at 18:20 +0100, Heinz-Josef Claes wrote: > > > I want to check if the following is possible: > > > > > > 1. transport a big block of data (several terabytes) physically from > > > location > > > A to location B (very long distance) via tapes (or disks). > > > (Location A and B use different storage technologies.) > > > > > > When the tapes arrive in location B, the block of data has changed in > > > location > > > A (a program / OS is running and storing data in it). > > > > > > 2. shutdown application / OS in location A, rsync the delta between > > > Location A > > > and B online, then restart the system in location B. > > > > > > (Perhaps step 2 has to be done multiple times.) > > > > Since the source and destination versions are practically certain to > > differ, --checksum would serve no purpose. See the man page description > > of --checksum. > > Don't understand what you mean. From 1. und 2., only a few percent of > the data will change, so the idea is to transfer the differences only. > Transferring the whole file online takes too long. > How to do this without check sums (either --checksum or --inbound)? Did you read the description of --checksum as I suggested? It is an alternative "quick check" for deciding whether a file needs to be transferred, which is not what you want. You're talking about the delta-transfer algorithm, which is on by default for remote runs and is controlled by a separate option, --(no-)whole-file. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync of STDIN to a file.
On Thu, 2009-11-19 at 13:14 +, Mark Young wrote:
> I'm hoping rsync could be told to capture STDIN and treat it as a
> single file that it would then sync with the named file at the remote
> end. Perhaps something like this:
>
> tar czf - --files-from $FILE_LIST | rsync - $HOST:${BACKUP_FILE}
This could probably be implemented as a simple extension of the current
--copy-devices functionality. My only concern is that if the
delta-transfer algorithm makes a mistake, rsync will want to try the
transfer again, but stdin can only be consumed once. This could be
fixed by hacking in support for forking a subprocess to generate the
source content, or by using as the source a FUSE-based filesystem that
exposes the virtual tar file.
Another option would be to use rdiff(1). It can read from stdin, and I
don't think it attempts to retry if it makes a mistake.
To expand upon Ryan's remarks, even if you have an old compressed
tarball on the destination, the delta-transfer algorithm might not find
any matches since compression introduces non-local effects. To solve
that, the "gzip-rsyncable" patch (to gzip) would be needed. It was
maintained with rsync for a while but was apparently discontinued around
3.0.0pre6.
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync of STDIN to a file.
On Thu, 2009-11-19 at 16:28 +, Mark Young wrote: > I tried an experiment to see how rsync coped with the tar compressed > files versus the uncompressed files. I took a .tgz from last week and > rsync'd it with last nights version. rsync achieved the transfer with > a 1.61 speed up. Specifically 172MB of the 262MB file was transferred. > I then tried the same thing with the uncompressed tar files. rsync > achieved the transfer with a 690 speed up. Specifically 5MB for the > 3.6GB file was transferred. Clearly if diskspace were not an issue > this would be by far the superior option. Obviously I could increase > the complexity of the overall solution with compress & uncompress jobs > in sync with the backup strategy, but I believe in keeping it as > simple as possible when it comes to backups. Right. The gzip --rsyncable option is designed for this case: files compressed with --rsyncable should delta-transfer almost as well as uncompressed files, at a minor penalty in the compression ratio. > rdiff is a new one on me. Very interesting and something I'll > experiment with. For our strategy of 7 daily backups and 12 monthly > backups I guess I'd need to use two rdiff destinations so I could > apply the --remove-older-than 1W and --remove-older-than 365D options > respectively. It might not suit my current situation as explained > above regarding storing a tarball versus a directory hierachy, but > it's definitely worth knowing about. I was talking about rdiff (a command-line interface to the delta-transfer algorithm), not rdiff-backup (the incremental backup tool). > Also I'd not heard of the --rsyncable option for tar. It's not there > on my Debian Etch system, nor in my cygwin tar. So I guess it's not in > common distributions by default. Still it sounds useful and is > something I will bear in mind as something to consider. The option applies to gzip, not tar. In fact, it looks like Fedora applies an updated version of the gzip-rsyncable patch to their gzip package. You can get their patch from: http://cvs.fedoraproject.org/viewvc/rpms/gzip/F-11/gzip-1.3.9-rsync.patch?view=co -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync of STDIN to a file.
On Thu, 2009-11-19 at 13:02 -0500, Eliot Moss wrote: > Among other things, gzip embeds the names of the files that it > zipped ... so why would you expect your two .gz files to be > exactly the same? You'll need to use the same name for the > tar files ... and even then, I bet gzip records date and > time ... Yes, that gzip behavior is pretty annoying. I wish it would just stick to its job of compression and not embed random metadata in its output; that's tar's job. (TODO: complain upstream.) In any case, the behavior can be turned off with the -n option. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync of STDIN to a file.
On Thu, 2009-11-19 at 17:29 +, Mark Young wrote:
> So if I understand you correctly you are suggesting that I modify my
> backup line to be something like this:
>
> tar czf - --files-from $FILE_LIST | gzip --rsyncable > ${BACKUP_FILE}
> or:
> GZIP="--rsyncable" tar cvf - --files-from $FILE_LIST > ${BACKUP_FILE}
>
> then
> rsync ${BACKUP_FILE} $HOST:${BACKUP_FILE}
Correct.
> I just tried some tests but didn't get the results I expected. I seem
> to be getting quite different files:
[...]
> $ tar cf m9.tar ./newdir
> $ tar cf m10.tar ./newdir
> $ sum -r m*tar
> 54744 3340 m10.tar
> 54744 3340 m9.tar
> $ gzip --rsyncable m9.tar
> $ gzip --rsyncable m10.tar
> $ sum -r m*tar
> 21783 2929 m9.tar.gz
> 30175 2929 m10.tar.gz
>
> It seems that gzip produces different output even given the same
> input. Not very rsync friendly.
As Eliot said, this is gzip's metadata-saving behavior, which I suggest
you turn off with "-n". Even so, it is not as harmful to delta-transfer
performance as you make it sound: it will only result in sending one
extra block over the network.
> I'm not sure I can see what I'd use the rdiff command for since rsync
> will do the delta-transfer cleverness for me.
Rdiff can read from stdin more easily than rsync can, as per your
original question.
--
Matt
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Output of --list-only?
On Fri, 2009-11-20 at 12:21 -0500, Benjamin R. Haskell wrote: > Am I overlooking something obvious, or is there really no way to specify > an output format for '--list-only' or 'rsync -a [SOURCE]'? There is no way. You can enter an enhancement request if you like. > In case anyone's curious or another method's better, the goal is to use > multiple '--exclude-from=FILE' files, but to do something other than > 'rsync' with the list of files that would be generated by 'rsync'. > rsync --dry-run --out-format=%n -i -a [SOURCE] /nonexistent/path/ Yes, that's what I did in the same situation. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync of STDIN to a file.
On Fri, 2009-11-20 at 17:35 +, Mark Young wrote: > Can you point me at any usage examples with rdiff. I've searched quite > a bit and apart from the cygwin man page and the "rdiff -?" usage > message I'm unable to find enough information for me to see how to use > it successfully. Yes, the man page is very sparse; it ought to include an example of the complete process. In the meantime, I'll provide one here. Starting from src/file and dest/file : (cd dest && rdiff signature file file.sig) cp dest/file.sig src/ (cd src && rdiff delta file.sig file file.delta) cp src/file.delta dest/ (cd dest && rdiff patch file file.delta file.new) Now dest/file.new is a copy of src/file . For more about how this works, see: http://rsync.samba.org/tech_report/node2.html For remote operation, the "cp" commands should be replaced with an appropriate network copy command, perhaps even rsync itself. > In particular I'm not sure I see how it would be used in my scenario. > Currently there's no local copy of the backup image file. See above. You would also pipe your tar command to "rdiff delta", replacing the "file" argument with "-". -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync of STDIN to a file.
On Fri, 2009-11-20 at 19:01 -0500, Matt McCutchen wrote: > On Fri, 2009-11-20 at 17:35 +, Mark Young wrote: > > Can you point me at any usage examples with rdiff. I've searched quite > > a bit and apart from the cygwin man page and the "rdiff -?" usage > > message I'm unable to find enough information for me to see how to use > > it successfully. > > Yes, the man page is very sparse; it ought to include an example of the > complete process. Filed upstream: https://sourceforge.net/tracker/?func=detail&aid=2901539&group_id=56125&atid=479441 -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
