3.0.0pre2: bookend breakage (2 different errors)
# The first error rsync: generator.c:1867: check_for_finished_files: Assertion `flist != ((void *)0)' failed. rsync: writefd_unbuffered failed to write 4092 bytes [receiver]: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(1493) [receiver=3.0.0pre2] # Sample command, obfuscated to protect the guilty rsync --archive --hard-links --no-motd \ --numeric-ids --sparse --verbose \ --link-dest $BACKUPPATH/hydrogen/monday/ \ --link-dest $BACKUPPATH/helium/monday/ \ --link-dest $BACKUPPATH/lithium/monday/ \ --link-dest $BACKUPPATH/beryllium/monday/ \ --link-dest $BACKUPPATH/oxygen/monday/ \ --link-dest $BACKUPPATH/fluorine/sunday/ \ --password-file $BACKUPPATH/fluorine/$BACKUPPASSWORDFILE \ --temp-dir $BACKUPPATH/fluorine/ \ rsync://[EMAIL PROTECTED]:$BACKUPPORT/$BACKUPMODULE/ \ $BACKUPPATH/fluorine/monday/ \ $BACKUPPATH/fluorine/monday.log # Background I do rotating backups of the entire 'running' fs on all my linux machines (ie: server data drives, ~/.ccache/, squid cache suchlike all excluded). Friday past, I installed rsync 3.0.0pre2 upgraded glibc-zoneinfo (slackware update that I finally got around to dealing with) on all my machines. The next set of backups (00:00 Saturday) immediately started throwing the above error, all in /usr/share/zoneinfo/. Interestingly, the first machine in the set barely made it 1/3rd of the way through this dir, second machine roughly 1/2 way, third machine further yet, fourth through sixth completed their backups successfully. The basic idea of my backup scheme is a 7 day rotating backup for each machine (hostname/weekday). Delete the obsolete backup for the current host, then --link-dest is used to link yesterday's set of the current host 'forward' (making 'cp -al' unnecessary) while also linking today's set of all other hosts 'sideways' to catch common updates. Simple, effective, it's worked for the past several years. You'd be amazed how little space 42 complete sets of backups take when they're all hardlinked together. As a quick'n'dirty test, I removed all but the 'link yesterday forward' --link-dest. Still threw the first error. When I added a 'cp -al' step, added --delete-after, removed all usage of --link-dest, it threw a _different_ error without actually getting a chance to transfer any updated files. Yay? # The second error Invalid file index: -101 (-1 - 0) with iflags 0 [receiver] rsync error: protocol incompatibility (code 2) at rsync.c(273) [receiver=3.0.0pre2] rsync: connection unexpectedly closed (21 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(596) [generator=3.0.0pre2] # Sample commands, obfuscated to protect the guilty cp -alf $BACKUPPATH/fluorine/sunday/. $BACKUPPATH/fluorine/monday/ rsync --archive --delete-after --hard-links --no-motd \ --numeric-ids --sparse --verbose \ --password-file $BACKUPPATH/fluorine/$BACKUPPASSWORDFILE \ --temp-dir $BACKUPPATH/fluorine/ \ rsync://[EMAIL PROTECTED]:$BACKUPPORT/$BACKUPMODULE/ \ $BACKUPPATH/fluorine/monday/ \ $BACKUPPATH/fluorine/monday.log I've undoubtedly left out some critical info, but I've been staring at this too long. More, if needed, later. Erik -- Failure is not an option. (It comes bundled with Windows.) If at first you don't succeed, redefine success. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[feature request] Modify --backup option to make backups on source side, too
Hello, Can you please modify the backup options to work together with --remove-source-files so I'm able to create a reliable move between hosts script? If --backup is specified, the files to be deleted should be renamed on the source side. If --backup-dir is specified, they should be moved there, etc. If this already works, please update the documentation accordingly :-) Thanks a lot, -- Aaron Optimizer Digulla a.k.a. Philmann Dark It's not the universe that's limited, it's our imagination. Follow me and I'll show you something beyond the limits. http://www.pdark.de/ -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: PATCH: preserve osx creation-date (was: Using pre2 for backing up a mac)
On Oct 15, 2007, at 1:55 AM, Wayne Davison wrote:
On Sun, Oct 14, 2007 at 08:14:57PM +0200, Wesley W. Terpstra wrote:
I've attached a patch which does this. Currently resource forks
and finder
info get placed into an extended attribute transparently by osx
(com.apple.{ResourceFork/FinderInfo}). This patch makes another
extended
attribute called com.apple.CreationDate.
Thanks! I've twiddled your patch a bit and checked it in as a file in
the patches dir (for now, at least): osx-create-time.diff
I changed the storing of the data to store the time value as 12
bytes of
binary data -- 8 for the time_t value (for future expansion) and 4 for
the nsec value (which always be 0 at present).
I fail to see the advantage of a binary version. You add concerns
about byte-order and make it unreadable to a sysadmin. (I also don't
like how the ACL stuff is binary) There was no problem with 8 byte
time_t vs. 4 byte values with the other format either as it would've
been good until year . If stat is textual, why not this? Saving a
few bytes is not a good reason ...
The name of the attribute was changed to com.apple.crtime96 (for
the moment) . Since it is not an official com.apple.* value, I
didn't want to use a name that Apple might choose in the future.
The name should probably go in the rsync.* namespace, but I'd need
to move the code for reading and writing the value out of lib/
sysxattrs.c into xattrs.c to do that (I believe).
Yeah, I suppose it should go in the rsync namespace too.
I also enabled the XATTR_NOFOLLOW option for {get,set}attrlist().
This option doesn't exist for these system calls. There is, however,
FSOPT_NOFOLLOW.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Symlinks in OS X (10.4.1)
On Oct 14, 2007, at 10:05 PM, [EMAIL PROTECTED] wrote: Actually, this isn't quite true. osx has no lchmod, but if you set a umask before creating the link, the permissions are set. So, as opposed to ignoring the permissions, perhaps we should teach rsync how to set them. I'll probably look at this as soon as I'm done writing a patch for preserving the creation date. I know about the missing lchmod in OS X, but I don't really see a need for it. Why can't a symlink just have a default permission of 0777? In case the symlink gets dereferenced the target's permissions have to be considered anyway. Of course it is cleaner to have them set properly, but if they are just ignored it won't hurt either. I believe in osx the symlink permissions have meaning (as opposed to linux). Eg: -rwxrwxrwx bar r--r-- foo - bar In linux, 'cat foo' will work, in osx, it will fail. So, this meta- data has meaning, and to just replace it with 0777 will not give you a functionally equal copy. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: PATCH: preserve osx creation-date (was: Using pre2 for backing up a mac)
On Oct 15, 2007, at 12:13 PM, Wesley W. Terpstra wrote:
On Oct 15, 2007, at 1:55 AM, Wayne Davison wrote:
The name of the attribute was changed to com.apple.crtime96 (for
the moment) . Since it is not an official com.apple.* value, I
didn't want to use a name that Apple might choose in the future.
The name should probably go in the rsync.* namespace, but I'd need
to move the code for reading and writing the value out of lib/
sysxattrs.c into xattrs.c to do that (I believe).
Yeah, I suppose it should go in the rsync namespace too.
I notice that rsync is starting to converge towards a particular use
of xattrs:
Copy meta-data that has an analogous concept on the remote host.
If -X is specified, copy anything that has no analogue into an
extended attribute.
In looking over the bsd flags patch, I notice that it is relatively
complicated and still doesn't move the data into an xattr on a system
which has no support.
I wonder if the right command-line options might be something like this:
-X ... enables copying of user extended attributes
--flags ... enables copying of bsd flags
--osx (or similar) ... enables copying of osx related meta-data
(ResourceFork/FinderInfo/CreationDate)
-A ... enables copying of ACL information
-M ... maps any unsupported meta-data into an extended attribute on
the remote side (in the rsync namespace)
-Q ... silently drop unsupported meta-data
I'm not sure if --flags and --osx should be implied by -p, but I
think so.
Then, use a pseudo-design pattern where every type of meta-data rsync
can get/set has do_{get,set}_{acls,flags,owner,group,...}. A platform
also provides a sys_supports_{acls,flags,...}(filename). This returns
true/false if the meta-data is supported for that file (filesystems
differ, as do platforms). Then, if the method could ever return true,
the platform must provide matching sys_{set,get,could_set}_
(filename) methods. could_set reports if there are sufficient
permissions for the operation to succeed.
A design pattern like this could be used to easily extend rsync
whenever new types of meta-data appear. For each piece of meta-data,
the do_{get,set} methods look like this:
do_set_X(file, ...):
if (sys_supports_X(file)) {
if (fake_super and NOT sys_could_set_X(file)) {
return set_xattr(file, X, ...)
} else {
return sys_set_X(file, ...)
}
} else {
if (-M) {
return set_xattr(file, X, ...)
} else if (! -Q) {
warning: meta-data X is unsupported on platform Y for file, use -
Q to silence this warning, or -M to map it into an xattr.
return
}
}
do_get_X:
if (sys_supports_X(file)) {
if (fake_super and NOT sys_could_set_X(file)) {
return get_xattr(file, X)
} else {
return sys_get_X(file)
}
} else {
if (-M) {
return get_xattr(file, X)
}
if (! -Q) {
if (exist_xattr(file, X)) warning: discarding meta-data X that
was stored in an xattr
}
return nothing_to_sync
}
Real implementations would be complicated by meta-data that is
partially convertible. For example, file permissions and ACLs. But
logically, from a user point of view, one wants the above behaviour.
What do you think?
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [feature request] Modify --backup option to make backups on source side, too
On 10/15/07, Aaron Digulla [EMAIL PROTECTED] wrote: Can you please modify the backup options to work together with --remove-source-files so I'm able to create a reliable move between hosts script? The patch patches/source-backup.diff in the source tree adds an option --source-backup to do this. You can compile your own copy of rsync containing the patch and then use that option. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync error: protocol incompatibility (code 2) at main.c(1385)
Please CC the list so people other than me can help you and so your messages will be available to future users. On 10/15/07, Mario Domgörgen [EMAIL PROTECTED] wrote: No, i don't run the client side command myself. The backup should start from the server, so i use the the second command to connect to the client with an unpriviledged user, that has a forced command in his authorized_keys file. The particular data sent by the rsync protocol depends heavily on the options used, so if the forced command doesn't match the remote command that rsync is trying to invoke, the protocol will get corrupted. I bet this is what caused the error. Forcing a particular --server command is brittle; instead, you should force an appropriately configured single-use daemon (my personal preference) or use the support/rrsync script in the source tree. See https://bugzilla.samba.org/show_bug.cgi?id=4163 . Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: SSH daemon questions
On 10/13/07, Alan Cheers [EMAIL PROTECTED] wrote: If you had multiple people making rsync backups over ssh wouldn't it be preferred to use the single-use daemons from a security standpoint? If multiple people use this method I would want to limit the chance of somebody being able to grab or overwrite somebody else's data. Which do you think is better for a multiuser setup? If the users have full shell accounts on the destination system (or you are willing to give them accounts), then each user can push backups to his/her own account over ssh and can't access anyone else's backups. A single-use daemon could optionally be used for the convenience of having the backups rotated automatically but would not be involved in the security model. If the users do not have shell accounts, then the daemon becomes important as a gatekeeper. The easiest setup would use a background (opposite of single-use) daemon with one module for each user. A user would authenticate to the daemon using a username/password configured in the daemon's secrets file and could then access only his/her own module according to the auth users setting in the rsyncd.conf. Note that the daemon protocol provides neither encryption nor integrity-checking for the connection, so if there is the potential for untrusted people to tap the connection, you should protect it somehow. One relatively easy way is to convert the daemon to a single-use daemon over ssh, create an authorized key that has a forced command that invokes the single-use daemon, and publish the authorized key for everyone to use. This way, any user can log in through ssh and know they have a secure connection to the daemon, but then he/she must authenticate to the daemon to gain access to modules. Note: If you intend to deploy rsync for a specific purpose (rather than just learn about it), it would help if you gave me the entire picture now instead of revealing it piece by piece. Let's bring this conversation onto the list in case the information is useful to others in the future. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [feature request] Modify --backup option to make backups on source side, too
On 10/15/07, Aaron Digulla [EMAIL PROTECTED] wrote: Thanks for the input. If this wasn't for a production system, I'd do that :-) Why don't you apply the patch the release version? Because the patch is a quick hack to serve a niche need, so the benefit of saving people who need source backups a few steps does not outweigh the cost of having it clutter up (and conceivably destabilize) the main version of rsync for everyone else. I think the model of a main version with the most important features plus distributed patches for various extra features is an excellent way to deliver a clean, stable rsync to the typical user but still have implementations of the extra features available when needed. In fact, beginning in September 2005, I used a copy of rsync including a patch for ACL support for nearly all copies on my computer. The patch was a complete hack and I really cared about ACLs, so I helped Wayne improve the patch to the point where it was just as clean and stable as the main code; then, in March 2007, Wayne elected to roll the patch into the main version. In the same way, I encourage you to test and begin using the source-backup patch now if it suits your needs; if you want the patch added to the main version, you can help make that more likely by improving it. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: PATCH: preserve osx creation-date (was: Using pre2 for backing up a mac)
This is a general topic I'm interested in, as I rely on rsync to back
up my mac osx files
to a remote file server. Unfortunately, the file server is a sun
workstation, and I don't know
if it supports xattr's, or large xattr's that arise from resource
forks, and I'm not sure if the sys admins
back up resource forks when they backup the file server to tape.
I hacked together a bit of code to deal with this ( http://
www.shoup.net/xattr ).
The idea is to generate shadow files that store all the extra meta
data on the file server.
To avoid hacking rsync internals, these shadow files are stored in a
shadow directory with the
same structure as the original.
What would be nice is if rsync supported this directly, with the
shadow files (with names appropriately tweaked)
stored in the same directory structure. This is essentially Apple's
solution for dealing with foreign file systems
that are mounted locally (e.g., NFS shares or local UFS volumes), but
this does not work at all
between mac osx and a remote file server accessed via ssh.
I've seen some rsync patches on the web that do just this, but it
would be much better if it were in the main rsync source.
-- Victor Shoup
On Oct 15, 2007, at 6:13 AM, Wesley W. Terpstra wrote:
On Oct 15, 2007, at 1:55 AM, Wayne Davison wrote:
On Sun, Oct 14, 2007 at 08:14:57PM +0200, Wesley W. Terpstra wrote:
I've attached a patch which does this. Currently resource forks
and finder
info get placed into an extended attribute transparently by osx
(com.apple.{ResourceFork/FinderInfo}). This patch makes another
extended
attribute called com.apple.CreationDate.
Thanks! I've twiddled your patch a bit and checked it in as a
file in
the patches dir (for now, at least): osx-create-time.diff
I changed the storing of the data to store the time value as 12
bytes of
binary data -- 8 for the time_t value (for future expansion) and 4
for
the nsec value (which always be 0 at present).
I fail to see the advantage of a binary version. You add concerns
about byte-order and make it unreadable to a sysadmin. (I also
don't like how the ACL stuff is binary) There was no problem with 8
byte time_t vs. 4 byte values with the other format either as it
would've been good until year . If stat is textual, why not
this? Saving a few bytes is not a good reason ...
The name of the attribute was changed to com.apple.crtime96 (for
the moment) . Since it is not an official com.apple.* value, I
didn't want to use a name that Apple might choose in the future.
The name should probably go in the rsync.* namespace, but I'd need
to move the code for reading and writing the value out of lib/
sysxattrs.c into xattrs.c to do that (I believe).
Yeah, I suppose it should go in the rsync namespace too.
I also enabled the XATTR_NOFOLLOW option for {get,set}attrlist().
This option doesn't exist for these system calls. There is,
however, FSOPT_NOFOLLOW.
--
To unsubscribe or change options: https://lists.samba.org/mailman/
listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-
questions.html
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: PATCH: preserve osx creation-date (was: Using pre2 for backing up a mac)
On Oct 15, 2007, at 5:07 PM, Victor Shoup wrote: What would be nice is if rsync supported this directly, with the shadow files (with names appropriately tweaked) stored in the same directory structure. This is essentially Apple's solution for dealing with foreign file systems that are mounted locally (e.g., NFS shares or local UFS volumes), but this does not work at all between mac osx and a remote file server accessed via ssh. The problem with doing this is that you've separated the meta-data from the file. I don't think rsync should do this itself. It introduces problems when other tools interact with your backup. It's very easy to lose the meta-data. Unfortunately, ext3 seems to have a very low limit on extended attributes (resource forks tend to be too big). XFS works great, but on a couple of my machines (one is an Infrant ReadyNAS) I can't avoid using ext3. For this scenario I've written a FUSE filesystem that simply redirects extended attributes into an sqlite3 database (which it keys by inode, so nothing breaks). This accomplishes what you wanted with your shadow files, but on the filesystem level it doesn't separate the meta-data from the file. It also means rsync can stick to doing just its own job. It seems Solaris has FUSE, so maybe this would work for you too... http://opensolaris.org/os/project/fuse/ -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--backup-dir or --link-dest?
Hi, To do incremental backups I am presently using --backup-dir=/backup/host/$TODAY and the full backup is written to /backup/host/current. With this system incremental directories only contain deleted files and current contains the full backup. Would it be more efficient to use --link-dest=/backup/host/$YESTERDAY and write the main backup to /backup/host/$TODAY ? That way each day incremental would contain the full daily snapshot, wouldn't it? What is the current best practice standard for incremental backups? Thanks, -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Symlinks in OS X (10.4.1)
On Mon, Oct 15, 2007 at 12:18:39PM +0200, Wesley W. Terpstra wrote: I believe in osx the symlink permissions have meaning (as opposed to linux). Is there a system call that allows their permissions to be changed after they've been created? (Since lchmod is lacking.) ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [feature request] Modify --backup option to make backups on source side, too
Matt McCutchen schrieb: In the same way, I encourage you to test and begin using the source-backup patch now if it suits your needs; if you want the patch added to the main version, you can help make that more likely by improving it. Thanks. I'll try to convince my bosses to see this as payment for using rsync :-) I just hope I can find a simple way to build rsync on Windows ... I guess I'll try with Cygwin since I already have that installed and I could build GhostPCL with it. Or do you recommend something else? This is for a production system which goes online Q1 2008. Should I use the latest stable version of rsync (2.6.9) or will there be a release of 3.0 until then? PS: Thanks for the new incremental-recursion algorithm. That one has bugged me for a long time! Syncing 150GB of build directories sure spent a lng time in the scanning stage :-) Best regards, -- Aaron Optimizer Digulla a.k.a. Philmann Dark It's not the universe that's limited, it's our imagination. Follow me and I'll show you something beyond the limits. http://www.pdark.de/ -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [feature request] Modify --backup option to make backups on source side, too
On 10/15/07, Aaron Digulla [EMAIL PROTECTED] wrote: Thanks. I'll try to convince my bosses to see this as payment for using rsync :-) I just hope I can find a simple way to build rsync on Windows ... I guess I'll try with Cygwin since I already have that installed and I could build GhostPCL with it. Or do you recommend something else? Rsync on Cygwin is very well supported; unfortunately, remote copies sometimes hang due to deficiencies in Cygwin. I recommend that you try Cygwin first and then, if the hang becomes a problem, Interix. This is for a production system which goes online Q1 2008. Should I use the latest stable version of rsync (2.6.9) or will there be a release of 3.0 until then? Rsync 3.0.0 is in pre-releases now. Chances are very good that the final release will be comfortably before Q1 2008. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Symlinks in OS X (10.4.1)
On 10/15/07, Wayne Davison [EMAIL PROTECTED] wrote: Is there a system call that allows their permissions to be changed after they've been created? (Since lchmod is lacking.) I don't think so, but as Wesley hinted, one can effectively change a symlink's permissions by deleting it, setting the umask appropriately, and recreating it. I think this behavior is useful enough to be provided by a distributed patch. It does break hard links; are there any other caveats? Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 4870] --link-dest results in errors for some paths containing spaces
https://bugzilla.samba.org/show_bug.cgi?id=4870 --- Comment #2 from [EMAIL PROTECTED] 2007-10-15 21:24 CST --- Wayne, what space are you talking about? I see a basis dir ../1, a source /Users/coppit/Documents/, and a destination rsync://[EMAIL PROTECTED]/Backups/coppit/0, none of which contains a space. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
To inc_recurse or not to inc_recurse? [Re: 3.0.0pre2: bookend breakage (2 different errors)]
On 10/15/07, Erik Jan Tromp [EMAIL PROTECTED] wrote: # The second error Invalid file index: -101 (-1 - 0) with iflags 0 [receiver] rsync error: protocol incompatibility (code 2) at rsync.c(273) [receiver=3.0.0pre2] rsync: connection unexpectedly closed (21 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(596) [generator=3.0.0pre2] # Sample commands, obfuscated to protect the guilty cp -alf $BACKUPPATH/fluorine/sunday/. $BACKUPPATH/fluorine/monday/ rsync [...] --delete-after [...] rsync://[EMAIL PROTECTED]:$BACKUPPORT/$BACKUPMODULE/ \ $BACKUPPATH/fluorine/monday/ [...] This is the same problem I ran into with --detect-renamed: the client disables incremental recursion because of --delete-after but doesn't send the option to the server, so the server has no idea that it is supposed to disable incremental recursion. The other options that can disable incremental recursion (see compat.c) may cause the same crash. I suppose server_options could be changed to send all of these options unconditionally. However, a more discreet and much more robust solution would be for the client and server to negotiate whether to incrementally recurse just after negotiating the protocol version. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
detect-renamed-lax.diff
On 10/13/07, Wayne Davison [EMAIL PROTECTED] committed: Added Files: detect-renamed-lax.diff Log Message: My version of Matt's --trust-rename patch. The option --detect-moved should be named --detect-moved-lax because it contains the lax behavior. --detect-moved would mean use a file with the same basename as an alternate basis. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: detect-renamed-lax.diff
On Mon, Oct 15, 2007 at 11:21:04PM -0400, Matt McCutchen wrote: The option --detect-moved should be named --detect-moved-lax because it contains the lax behavior. I don't think so. Since the name matches (just not where it lives) it's an extra measure of accuracy. However, since none of these options are going to make it into a release of rsync, so you can name the option anything you like... ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: To inc_recurse or not to inc_recurse? [Re: 3.0.0pre2: bookend breakage (2 different errors)]
On Mon, Oct 15, 2007 at 11:15:34PM -0400, Matt McCutchen wrote: However, a more discreet and much more robust solution would be for the client and server to negotiate whether to incrementally recurse just after negotiating the protocol version. Yeah, I agree that it is better for the client to explicitly tell the server what is going on (and allows a batch file to indicate what is happening too). That way there is no confusion. It also allows for future expansion in certain situations -- e.g. I can imagine making a future version of --prune-empty-dirs and/or --delay-updates compatible with inc-recursion, and this will allow a more modern rsync to try to tell a remote receiver to use that option and stay in inc_recurse mode, and the receiver can say no, I don't support that. The CVS version (and latest nightly) have this fixed. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 4870] --link-dest results in errors for some paths containing spaces
https://bugzilla.samba.org/show_bug.cgi?id=4870 --- Comment #3 from [EMAIL PROTECTED] 2007-10-15 23:22 CST --- Sorry, I thought that the mention of a space was to indicate that the space was in the command-line arg. Spaces in names read from directories don't make any difference to rsync. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Symlinks in OS X (10.4.1)
On Oct 15, 2007, at 8:19 PM, Wayne Davison wrote: On Mon, Oct 15, 2007 at 12:18:39PM +0200, Wesley W. Terpstra wrote: I believe in osx the symlink permissions have meaning (as opposed to linux). Is there a system call that allows their permissions to be changed after they've been created? (Since lchmod is lacking.) I was about to say no, but then I saw this: setattrlist + ATTR_CMN_ACCESSMASK I'll check tomorrow if one can write lchmod for osx using this. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: To inc_recurse or not to inc_recurse? [Re: 3.0.0pre2: bookend breakage (2 different errors)]
On 10/16/07, Wayne Davison [EMAIL PROTECTED] wrote: It also allows for future expansion in certain situations -- e.g. I can imagine making a future version of --prune-empty-dirs and/or --delay-updates compatible with inc-recursion, and this will allow a more modern rsync to try to tell a remote receiver to use that option and stay in inc_recurse mode, and the receiver can say no, I don't support that. In this situation, automatically disabling incremental recursion would be better than exiting with an error that really isn't the user's fault. (That's why I said negotiate.) Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: To inc_recurse or not to inc_recurse? [Re: 3.0.0pre2: bookend breakage (2 different errors)]
On 10/16/07, Wayne Davison [EMAIL PROTECTED] wrote:
Yeah, I agree that it is better for the client to explicitly tell the
server what is going on (and allows a batch file to indicate what is
happening too).
Now you could revert the unconditional sending of --detect-renamed in
detect-renamed.diff; the new detect-renamed.diff is attached.
Matt
This patch adds the --detect-renamed option which makes rsync notice files
that either (1) match in size modify-time (plus the basename, if possible)
or (2) match in size checksum (when --checksum was also specified) and use
each match as an alternate basis file to speed up the transfer.
The algorithm attempts to scan the receiving-side's files in an efficient
manner. If --delete[-before] is enabled, we'll take advantage of the
pre-transfer delete pass to prepare any alternate-basis-file matches we
might find. If --delete-before is not enabled, rsync does the rename scan
during the regular file-sending scan (scanning each directory right before
the generator starts updating files from that dir). In this latter mode,
rsync might delay the updating of a file (if no alternate-basis match was
yet found) until the full scan of the receiving side is complete, at which
point any delayed files are processed.
I chose to hard-link the alternate-basis files into a .~tmp~ subdir that
takes advantage of rsync's pre-existing partial-dir logic. This uses less
memory than trying to keep track of the matches internally, and also allows
any deletions or file-updates to occur normally without interfering with
these alternate-basis discoveries.
To use this patch, run these commands for a successful build:
patch -p1 patches/detect-renamed.diff
./configure (optional if already run)
make
TODO:
We need to never return a match from fattr_find() that has a basis
file. This will ensure that we don't try to give a renamed file to
a file that can't use it, while missing out on giving it to a file
that could use it.
--- old/compat.c
+++ new/compat.c
@@ -41,6 +41,7 @@ extern int checksum_seed;
extern int basis_dir_cnt;
extern int prune_empty_dirs;
extern int protocol_version;
+extern int detect_renamed;
extern int protect_args;
extern int preserve_uid;
extern int preserve_gid;
@@ -218,7 +219,7 @@ void setup_protocol(int f_out,int f_in)
} else if (protocol_version = 30) {
if (recurse allow_inc_recurse
!delete_before !delete_after !delay_updates
- !use_qsort !prune_empty_dirs)
+ !use_qsort !prune_empty_dirs !detect_renamed)
inc_recurse = 1;
if (am_server || read_batch) {
int i_r = read_byte(f_in);
--- old/flist.c
+++ new/flist.c
@@ -61,6 +61,7 @@ extern int non_perishable_cnt;
extern int prune_empty_dirs;
extern int copy_links;
extern int copy_unsafe_links;
+extern int detect_renamed;
extern int protocol_version;
extern int sanitize_paths;
extern struct stats stats;
@@ -113,6 +114,8 @@ static int64 tmp_dev, tmp_ino;
#endif
static char tmp_sum[MAX_DIGEST_LEN];
+struct file_list the_fattr_list;
+
static char empty_sum[MAX_DIGEST_LEN];
static int flist_count_offset; /* for --delete --progress */
static int dir_count = 0;
@@ -252,6 +255,45 @@ static int is_excluded(char *fname, int
return 0;
}
+static int fattr_compare(struct file_struct **file1, struct file_struct **file2)
+{
+ struct file_struct *f1 = *file1;
+ struct file_struct *f2 = *file2;
+ int64 len1 = F_LENGTH(f1), len2 = F_LENGTH(f2);
+ int diff;
+
+ if (!f1-basename || !S_ISREG(f1-mode) || !len1) {
+ if (!f2-basename || !S_ISREG(f2-mode) || !len2)
+ return 0;
+ return 1;
+ }
+ if (!f2-basename || !S_ISREG(f2-mode) || !len2)
+ return -1;
+
+ /* Don't use diff for values that are longer than an int. */
+ if (len1 != len2)
+ return len1 len2 ? -1 : 1;
+
+ if (always_checksum) {
+ diff = u_memcmp(F_SUM(f1), F_SUM(f2), checksum_len);
+ if (diff)
+ return diff;
+ } else if (f1-modtime != f2-modtime)
+ return f1-modtime f2-modtime ? -1 : 1;
+
+ diff = u_strcmp(f1-basename, f2-basename);
+ if (diff)
+ return diff;
+
+ if (f1-dirname == f2-dirname)
+ return 0;
+ if (!f1-dirname)
+ return -1;
+ if (!f2-dirname)
+ return 1;
+ return u_strcmp(f1-dirname, f2-dirname);
+}
+
static void send_directory(int f, struct file_list *flist,
char *fbuf, int len, int flags);
@@ -2154,6 +2196,25 @@ struct file_list *recv_file_list(int f)
clean_flist(flist, relative_paths);
+ if (detect_renamed) {
+ int j = flist-used;
+ the_fattr_list.used = j;
+ the_fattr_list.files = new_array(struct file_struct *, j);
+ if (!the_fattr_list.files)
+ out_of_memory(recv_file_list);
+ memcpy(the_fattr_list.files, flist-files,
+ j * sizeof (struct file_struct *));
+ qsort(the_fattr_list.files, j,
+ sizeof the_fattr_list.files[0], (int (*)())fattr_compare);
+ the_fattr_list.low = 0;
+ while (j-- 0) {
+ struct file_struct *fp = the_fattr_list.files[j];
+ if (fp-basename S_ISREG(fp-mode)
DO NOT REPLY [Bug 4870] --link-dest results in errors for some paths containing spaces
https://bugzilla.samba.org/show_bug.cgi?id=4870 --- Comment #4 from [EMAIL PROTECTED] 2007-10-15 23:50 CST --- Then, something else is going on here, and this report deserves further investigation. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: To inc_recurse or not to inc_recurse? [Re: 3.0.0pre2: bookend breakage (2 different errors)]
On 10/16/07, Matt McCutchen [EMAIL PROTECTED] wrote: (That's why I said negotiate.) One more thing I want to point out in case you haven't already thought of it. Once two-way negotiation is in place, each side should refuse incremental recursion only if it can't fulfill its *own* duties under incremental recursion; the sides should not second-guess each other. This way, e.g., incremental recursion can be used on a push with --delay-updates if the server supports the combination, regardless of whether the client would if it were the receiver. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Tue Oct 16 04:08:15 2007 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv18378 Modified Files: rsync.h Log Message: Incremented the SUBPROTOCOL_VERSION. Revisions: rsync.h 1.370 = 1.371 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.h?r1=1.370r2=1.371 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Tue Oct 16 04:08:18 2007 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv18402 Modified Files: compat.c Log Message: - Send a flag from the client to the server to indicate if the protocol is going to run in inc_recurse mode or not. - Verify that the options we received (as a server or as a batch-reader) are compatible with the requested inc_recurse mode. Revisions: compat.c1.65 = 1.66 http://www.samba.org/cgi-bin/cvsweb/rsync/compat.c?r1=1.65r2=1.66 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Tue Oct 16 04:08:21 2007 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv18425 Modified Files: io.c Log Message: Write out the inc_recurse flag into a protocol-30 batch file. Revisions: io.c1.248 = 1.249 http://www.samba.org/cgi-bin/cvsweb/rsync/io.c?r1=1.248r2=1.249 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Tue Oct 16 04:08:25 2007 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv18445 Modified Files: options.c Log Message: We don't need to send --no-i-r anymore. Revisions: options.c 1.413 = 1.414 http://www.samba.org/cgi-bin/cvsweb/rsync/options.c?r1=1.413r2=1.414 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/patches
Date: Tue Oct 16 05:11:08 2007 Author: wayned Update of /data/cvs/rsync/patches In directory dp.samba.org:/tmp/cvs-serv12283 Modified Files: osx-create-time.diff Log Message: Changed 2 uses of XATTR_NOFOLLOW to FSOPT_NOFOLLOW. Revisions: osx-create-time.diff1.1 = 1.2 http://www.samba.org/cgi-bin/cvsweb/rsync/patches/osx-create-time.diff?r1=1.1r2=1.2 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
