Re: [librsync-devel] librsync and rsync vulnerability to maliciously crafted data. was Re: MD4 checksum_seed
On Thu, 2004-04-08 at 12:36, Martin Pool wrote: > On 5 Apr 2004, Donovan Baarda <[EMAIL PROTECTED]> wrote: > > > librsync needs a whole file checksum. Without it, it silently fails for > > case 1), 3), and 4). > > Yes, a whole-file checksum should be used with it. Presumably > something stronger than md4 like SHA-1. md4 is probably good enough for most applications. I know it's not as secure as others, but when you take into account the requirement that the signature match as well, compromising it becomes much more complicated. > I think the only question is whether this should be done internally in > librsync, or as a separate process. I can see arguments either way. > > In some cases you might prefer to actually store an signed signature > using something like GPG. Yeah... good point. The rdiff program should probably use a whole file md4sum though. > > librsync could benefit from a random checksum_seed. It would need to be > > included in the signature. Without it librsync is vulnerable to cases 1) > > and 3). > > Random with respect to what? I think it would be nice if repeatedly > summing identicaly files gave identical signatures. Maybe it can vary > depending on only the input data... The problem is repeatability is what makes it vulnerable. If the content fully determines the signature, the content can be crafted to produce a particular signature. It is relatively easy to calculate two different blocks with the same blocksum if the checksum_seed is known. librsync could be modified to detect when two blocks had the same blocksum in a signature, and permutate the checksum seed in a deterministic way to try repeatedly until a signature without collisions is found. This would give repeatable signatures and protect against case 1), but not case 3). It would also allow crafted files that force m/2 recalculations of the signature. I think I've just realised what you were getting at; if the checksum_seed is based on something like the whole file md4sum, it becomes repeatable, but unpredictable. You can't manipulate individual blocks without it affecting every other blocksum, but the signature for the same file is always the same. Nice :-) This would fit in nicely with adding a whole-file checksum to the signature... generate the wholefile md4sum, and use that as the starting "seed" for the individual blocksums. The wholefile sum becomes the seed. This doesn't allow for "permutating" the seed if it happens to result in blocksum clashes. However, given that the probability if this happening is pretty astronomical (when using the wholefile sum seed) and will be caught by a whole-file checksum miss-match, do we care? It is far more likely to get false block matches when calculating the delta (because sums are calculated at every byte boundary, not just at block boundaries). More thoughts on using the wholefile sum as the seed; at first I thought this would still be vulnerable to case 3). Using a any single block as the original file and trying to find any block that matches means you still have "birthday algorithm" numbers of blocks to check (ie 2^(n/2)). However, each block "comparison" requires the recalculation of the "target" blocksum using the "original" blocksum as the seed, resulting in non-birthday algorithm number of checksum calculations (ie, 2^n). I've Cc'd this to the rsync list because they might get some ideas from it. -- Donovan Baarda <[EMAIL PROTECTED]> http://minkirri.apana.org.au/~abo/ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --suffix problem - possibly bug?
That *was* seriously impressive. Thanks Wayne. =) Steve W. Ingram wrote: 50 minutes for investigation *and* a patch!? Holy cow! Yeah, Mr Gates, that's the advantage of companies like yours; the support of your software. I mean, you just don't know where to get support on free software and what you do get, is er... oh hang on a minute! Amazing Wayne, just amazing. :) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --suffix problem - possibly bug?
So, just to clarify - this is a bug that needs fixing only on the server
side, not the client side, correct? (Important to me because I'd just
as druther not have to compile the cygwin side of the house if I don't
have to.)
On Wed, Apr 07, 2004 at 06:13:31PM -0400, Jim Salter wrote:
and either way, I wind up with a tilde as a suffix on all the files
moved into the backup directory at time of synchronization. According
Turns out to be a problem caused by the fact that a daemon goes through
the option-parsing code twice -- once when it first starts (and figures
out that it is a daemon), and again when it is parsing the options sent
from the connecting client. Since a client doesn't bother to send the
--suffix option if it is specifying a default value, the daemon server
ends up with the wrong default -- the one it decided on the first time
through the option-parsing code. This will be fixed in 2.6.1 (patch
appended).
Thanks for the report!
..wayne..
--- options.c 31 Mar 2004 17:02:22 - 1.143
+++ options.c 7 Apr 2004 23:04:48 -
@@ -688,6 +688,12 @@ int parse_arguments(int *argc, const cha
files_from = alloc_sanitize_path(files_from, curr_dir);
}
+ if (daemon_opt) {
+ daemon_opt = 0;
+ am_daemon = 1;
+ return 1;
+ }
+
if (!backup_suffix)
backup_suffix = backup_dir ? "" : BACKUP_SUFFIX;
backup_suffix_len = strlen(backup_suffix);
@@ -749,9 +755,6 @@ int parse_arguments(int *argc, const cha
}
}
}
-
- if (daemon_opt)
- am_daemon = 1;
return 1;
}
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync from solaris to cygwin on xp, certain files always copied
On Wed, Apr 07, 2004 at 11:25:15PM +, Kelly Felkins wrote: > 1) How can I tell if the file is being copied? Is there a way to ask > rsync to say "file x copied". Use -v for verbose output. > 2) How can I diagnose why these few files are being copied, and not others? Use -n to tell rsync not to really do any file updates. You can then look at the output of the -v option and try to figure out why a file is listed. Hint: rsync compares the file size and the timestamp by default, so I would assume that one of those items is different on the file(s) that it is wanting to update. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync from solaris to cygwin on xp, certain files always copied
Kelly Felkins yahoo.com> writes: > > I'm rsyncing a directory on a linux box to a win xp pro with cygwin... Opps. I meant to say: I'm rsyncing a directory on a *solaris* box to a win xp pro with cygwin... -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync is slowing down
On Wed, Apr 07, 2004 at 06:44:35PM -0500, Phil Howard wrote: > How would the sending side know what directories are "old" for a > given receiver? I didn't see anything about multiple receivers in your initial email, but it's not hard to come up with an optimizing strategy that allows more than a single one. [For instance, maintain a summary file for each archive-hierarchy delineation that exists, and if any of the summary files get updated during the initial slurp of data (which would exclude all archived data but not the summary files), start a second transfer that updates the relevant archive dir(s).] The big question is, should rsync get some kind of optimizing algorithm that expects/updates external data for the purposes of optimizing the filesystem search, or should this be something that exists outside of rsync. The current rsync algorithm is not made for the kind of short- circuiting of directory descent that this optimization would require, so I would not attempt to shoe-horn it into it. Something like this would work a lot better with a more modern protocol that processed each directory in succession rather than the whole tree all at once. There has been a good bit of talk (and even a little work) on designing an rsync successor, but nothing is forthcoming at the moment. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync is slowing down
On Sat, Apr 03, 2004 at 12:23:59PM -0800, Wayne Davison wrote: | You can implement such optimizations on top of rsync using either | excludes or the --files-from option. For instance, if the sending | side maintained an exclude file of old directories that didn't need | to be transferred, you could write a script that would look for | updated items and remove the appropriate exclusion. An exclude list | would have to be grabbed first from the remote side before it could | be used, though. How would the sending side know what directories are "old" for a given receiver? One receiving side may run their update today for an old directory that had one file changed. But another receiving side may not run its update for a few more days or even weeks. This sounds like the sending side needs to keep track of what each different receiver has or doesn't have. That's what I used to do before rsync. -- - | Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ | | (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ | - -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --suffix problem - possibly bug?
50 minutes for investigation *and* a patch!? Holy cow! Yeah, Mr Gates, that's the advantage of companies like yours; the support of your software. I mean, you just don't know where to get support on free software and what you do get, is er... oh hang on a minute! Amazing Wayne, just amazing. :) steve ---Original Message--- > From: Wayne Davison <[EMAIL PROTECTED]> > Subject: Re: --suffix problem - possibly bug? > Sent: 07 Apr 2004 23:07:24 > > On Wed, Apr 07, 2004 at 06:13:31PM -0400, Jim Salter wrote: > > and either way, I wind up with a tilde as a suffix on all the files > > moved into the backup directory at time of synchronization. According > > Turns out to be a problem caused by the fact that a daemon goes through > the option-parsing code twice -- once when it first starts (and figures > out that it is a daemon), and again when it is parsing the options sent > from the connecting client. Since a client doesn't bother to send the > --suffix option if it is specifying a default value, the daemon server > ends up with the wrong default -- the one it decided on the first time > through the option-parsing code. This will be fixed in 2.6.1 (patch > appended). > > Thanks for the report! > > ..wayne.. > -- > To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html ---Original Message--- -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync from solaris to cygwin on xp, certain files always copied
I'm rsyncing a directory on a linux box to a win xp pro with cygwin. Some of the files are always copied, or so it appears by the progress report and the fact that a dot version of the file is created and grows to the full size before completion. Note, the source file is not changing - it is a static file sitting in a directory and is not modified. Nor is the corresponding file in the target directory. The files that are always copied are relatively large - one of them is 1.8 GB. 1) How can I tell if the file is being copied? Is there a way to ask rsync to say "file x copied". 2) How can I diagnose why these few files are being copied, and not others? This is my latest command line: rsync -ae ssh --modify-window=5 --progress --delete --relative --exclude '.snapshot' sunbox:/home/kfelkins /backups As you can see, I've tried the modify-window option. I'm also using archive. Rsync is a great tool. Thanks for your help. -Kelly -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --suffix problem - possibly bug?
On Wed, Apr 07, 2004 at 06:13:31PM -0400, Jim Salter wrote:
> and either way, I wind up with a tilde as a suffix on all the files
> moved into the backup directory at time of synchronization. According
Turns out to be a problem caused by the fact that a daemon goes through
the option-parsing code twice -- once when it first starts (and figures
out that it is a daemon), and again when it is parsing the options sent
from the connecting client. Since a client doesn't bother to send the
--suffix option if it is specifying a default value, the daemon server
ends up with the wrong default -- the one it decided on the first time
through the option-parsing code. This will be fixed in 2.6.1 (patch
appended).
Thanks for the report!
..wayne..
--- options.c 31 Mar 2004 17:02:22 - 1.143
+++ options.c 7 Apr 2004 23:04:48 -
@@ -688,6 +688,12 @@ int parse_arguments(int *argc, const cha
files_from = alloc_sanitize_path(files_from, curr_dir);
}
+ if (daemon_opt) {
+ daemon_opt = 0;
+ am_daemon = 1;
+ return 1;
+ }
+
if (!backup_suffix)
backup_suffix = backup_dir ? "" : BACKUP_SUFFIX;
backup_suffix_len = strlen(backup_suffix);
@@ -749,9 +755,6 @@ int parse_arguments(int *argc, const cha
}
}
}
-
- if (daemon_opt)
- am_daemon = 1;
return 1;
}
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Q: --one-file-system and nested file systems
On Wed, Apr 07, 2004 at 02:47:16PM -0700, Wayne Davison wrote: > the -x option only affects the reading of the files on the source > machine. Correction: this is an inaccurate statement when using --delete; the hierarchy of files created on the receiving machine that is used to figure out what needs to be deleted is also constrained by the -x option. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--suffix problem - possibly bug?
I'm having an issue with the --suffix and -b flags - I've tried the following commands: rsync -avbr --backup-dir=/rsyncbackup rsynctest [EMAIL PROTECTED]::module rsync -avbr --backup-dir=/rsyncbackup --suffix="" rsynctest [EMAIL PROTECTED]::module and either way, I wind up with a tilde as a suffix on all the files moved into the backup directory at time of synchronization. According to the man page, if --backup-dir is specified, the default suffix should be a null character - but that's not what I'm discovering; I'm discovering that I CANNOT find a way to specify no suffix. I can successfully specify a DIFFERENT suffix - such as --suffix=.rsync - but I can't manage to specify no suffix at all. For reference, I'm issuing the rsync commands from a Windows 2000 laptop using cwrsync (based on rsync 2.6.0), and the server on the other end is a FreeBSD 5.1 machine running rsync 2.6.0 from the ports tree. Thanks ... Jim Salter JRS Systems -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Q: --one-file-system and nested file systems
On Wed, Apr 07, 2004 at 03:38:28PM -0400, Brian McEntire wrote: > * Is there a way for --one-file-system to know on dest side not to delete > files that are not part of the target file system on the src side? No, that's not currently possible -- the -x option only affects the reading of the files on the source machine. However, you can include all the source filesystems as args in a single copy command and it will enforce the single-filesystem (inode-based) restriction separately for each arg you specify: rsync -avxR -e ssh --numeric-ids --delete \ --exclude-from=/backups/control/all.exclude \ [EMAIL PROTECTED]:'/ /usr /usr/local /usr/local/apache' /backups/B As long as that doesn't exhaust your machine's memory, it should work fine. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Q: --one-file-system and nested file systems
Greetings rsync gurus! I'm trying to run rsync from machine A to backup files on machine B. I want the files to be copied into a directory -- /backups/B -- on machine A and mimic the directory structure on machine B. Machine B has the following file systems (and some NFS mounts I don't want): / /boot /usr /usr/local /usr/local/apache The problem comes when I run rsync with --one-file-system against the /usr file system on the remote machine. Rsync deletes all the files in and below /backups/B/usr/local on A as it rsyncs B:/usr I think I can work around this behavior by tuning the --exclude= pattern when I rsync against /usr/local and /usr/local/apache, but I was wondering if there is a better way. * Is there a way for --one-file-system to know on dest side not to delete files that are not part of the target file system on the src side? Example commands and details: First step works okay, backs up B:/ to A:/backups/B rsync -av -e ssh --one-file-system --numeric-ids --relative --delete \ --exclude-from=/backups/control/all.exclude [EMAIL PROTECTED]:/ . This is where it stops working as desired, rsync deletes everything under /backups/BB/usr/local on A while trying to rsync /usr from B -- rsync -av -e ssh --one-file-system --numeric-ids --relative --delete \ --exclude-from=/backups/control/all.exclude [EMAIL PROTECTED]:/usr . This also doesn't work as desired, it deletes everything under /backups/B/usr/local/apache -- rsync -av -e ssh --one-file-system --numeric-ids --relative --delete \ --exclude-from=/backups/control/all.exclude [EMAIL PROTECTED]:/usr/local . Finally, this works fine, it is the deepest file system on B: rsync -av -e ssh --one-file-system --numeric-ids --relative --delete \ --exclude-from=/backups/control/all.exclude [EMAIL PROTECTED]:/usr/local/apache . I've tried fiddling with trailing slashes on the src spec, tried w/wo --relative, and modifying the dest spec. I was not able to achieve the result I wanted and I'm pretty sure it's because the files on the remote side are not part of the remote file system but they are part of the dest file system and so are deleted by rsync. I'd much rather do this in a "pull" configuration similar to the commands given above than "pushing" the files... if possible. Any ideas? Did I miss something simple? (I hope not! :-) -Brian -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync new feature addition
Hello, I need to do a hack into rsync code --- suppose I am rsyncing two dirs A <=> B A |a |b -> link1 |c -> link2 |junk B |a |c -> link3 |b -> ./site_local/b |site_local/ what I need is - after running - $> rsync -avz --exclude="site_local" --delete A/ B/ link "b" should not change, but link "c" should The idea is to preserve all links in receiving area that are pointing to "site_local" I tried with some hacks in flist.c and exclude.c, but no success. Would be great if you give some help. Thanks, Ankur _ Ankur Chaplot Desing Automation Engineer Intel(r) Technologies India Pvt. Ltd. 136, Airport Road, Bangalore 5600017 ^-^ Direct: 91 80 25076517 iNet: 8-465-6517 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
