Regarding ownership !!

[Lakshminarayanan Radhakrishnan]

Dear Mr.Tim,

Options used in rsync command in our system:
rsync --verbose --recursive --update --delete
--group --owner --times --perm

Eventhough i am using "-owner" option while synching to the mate system.
The owner ship from the  System A is not restored in the System B.
( System B is destination ).

Why the ownership of System A is not restored in System B ?

( eg. )  file in system A :rwxr--r--  lakshmi  comp a.c
 file in system B:
 becomes,  rwxr--r--  nobody   nobody  a.c

Is there any option to restore the permission ?

thanks,
Lakshmi



[EMAIL PROTECTED] wrote:

> Glad you've got it going.  Performance depends on the particular version
> of rsync, options used, network, dasd, dasd interface (nfs, samba, direct
> (SCSI (version), IDE, FC)), Operating systems, memory, processors, other
> load, network traffic
>
> I actually get pretty cruddy results with rsync in my application, which
> involves mirroring a NAS device containing roughly 2M files in 102Mb, over
> a wan, using solaris to run rsync, forced, by a flaw in the NAS nfs unlink
> implementation which reorders unlinks, to use NFS2, which exposes the
> solaris mtime bug.  If I try to do it all, it takes about 2 days to grow
> to around 3Gb in memory, then crashes (there's plenty left over), so i
> can't do the whole thing in one swipe, which means hard links are not
> propogated, and deletions can be orphaned by the list generator (the
> script i use to break the jobs into chunks rsync can handle).
> People running against directly-attached dasd, or fast DFS, and doing
> reasonable-sized jobs, get really good speeds, and great efficiency.
>
>  I've had to write my own replacement, which is about to go into
> production, but I still love rsync.  Within its limitations, it's a
> superior tool.
>
> Tim Conway
> [EMAIL PROTECTED]
> 303.682.4917
> Philips Semiconductor - Longmont TC
> 1880 Industrial Circle, Suite D
> Longmont, CO 80501
> Available via SameTime Connect within Philips, n9hmg on AIM
> perl -e 'print pack(,
> 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970),
> ".\n" '
> "There are some who call me Tim?"
>
> Lakshminarayanan Radhakrishnan <[EMAIL PROTECTED]>
> Sent by: [EMAIL PROTECTED]
> 02/05/2002 08:57 AM
>
>
> To: Tim Conway/LMT/SC/[EMAIL PROTECTED]
> cc: [EMAIL PROTECTED]
> [EMAIL PROTECTED]
> [EMAIL PROTECTED]
> Subject:Re: Doubt in Rsync !!
> Classification:
>
> Dear Mr.Tim,
> Thank you very much for your valuable suggestions.
> Now I am able to mirror the set  of files from one system to  another
> system which are on the net.
> Yesterday, I calculated it is mirroring  188MB file in 63 sec from one side to 
> another side.
> Good performance.  Is anywhere  the performance  about rsync is mentioned
> in terms of  file size / time.
> I am going to use this rsync command in script continously to mirror the
> two systems always.
> Once again thanks,
>   regards,
>   laks
>
> [EMAIL PROTECTED] wrote:
> I just played back your mail in my head, and realized that you mentioned
> the rsync server.  I read your command, from which it was plain that you
> were NOT trying to contact a rsync server, and gave instructions based on
> that.  In case you were trying to contact a rsync server (rsyncd), I
> suggest you read the man pages for both rsync and rsyncd.conf.  the rsync
> manpage explains how to invoke rsync to have it be a server, and the
> rsyncd.conf manpage explains how to set up the required configuration
> file.  The rsync manpage also explains how to invoke rsync to CALL a
> server (as opposed to starting a temporary process via an external
> transport to act as your remote server, as your commandline showed).
> Direct consultation of the documentation which Tridge, Martin, Dave, and
> everybody else has put so much work into, can cover the broad
> possibilities with much less latency than an email list.
> Tim Conway
> [EMAIL PROTECTED]
> 303.682.4917
> Philips Semiconductor - Longmont TC
> 1880 Industrial Circle, Suite D
> Longmont, CO 80501
> Available via SameTime Connect within Philips, n9hmg on AIM
> perl -e 'print pack(,
> 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970),
> ".\n" '
> "There are some who call me Tim?"
> Tim Conway
> 02/04/2002 08:33 AM
> To: [EMAIL PROTECTED]
> cc: Lakshminarayanan Radhakrishnan
> <[EMAIL PROTECTED]>
> [EMAIL PROTECTED]
> [EMAIL PROTECTED]
> Subject:Re: Doubt in Rsync !!
> Classification: Unclassified
> Ok:  You're using an external transport (rsh, unless you've defined
> RSYNC_RSH as something else(probably ssh)).  First thing to check is
> whether you can rsh to destinationmachine.  See what happens if you do
> "rsh destinationmachine uname -a".  Does this report back the information
> for destinationmachine, or does it give you "Permission denied".  

Re: rsync is slowing down

[Tim Conway]

I wrote a tool to get around a similar problem.  In my case, I had to bail 
on rsync altogether... the hardware was just too unreliable and limited in 
resources.
I've BCCd the current custodian of that tool, who was at least at one time 
a member of this list.  If my old company has no objection to the sharing 
if my IP and he gets a moment to come up for air amid his drowning in both 
his and my work, maybe he can post it.  If you're going to drive rsync 
with it, you'll need some mods, as currently, it uses packetized tar 
heck, you'll need mods anyway, as it's got a lot of bug workarounds to get 
along with the incompetent Maxtor NAS devices, and the code is so ugly, 
you may have to rewrite it anyway just to avoid hurting your eyes.

Tim Conway
Unix System Administration
Contractor - IBM Global Services
desk:3032734776
[EMAIL PROTECTED]




Phil Howard <[EMAIL PROTECTED]> 
Sent by: [EMAIL PROTECTED]
04/07/2004 05:44 PM

To
[EMAIL PROTECTED]
cc

Subject
Re: rsync is slowing down






On Sat, Apr 03, 2004 at 12:23:59PM -0800, Wayne Davison wrote:

| You can implement such optimizations on top of rsync using either
| excludes or the --files-from option.  For instance, if the sending
| side maintained an exclude file of old directories that didn't need
| to be transferred, you could write a script that would look for
| updated items and remove the appropriate exclusion.  An exclude list
| would have to be grabbed first from the remote side before it could
| be used, though.

How would the sending side know what directories are "old" for a
given receiver?  One receiving side may run their update today for
an old directory that had one file changed.  But another receiving
side may not run its update for a few more days or even weeks.

This sounds like the sending side needs to keep track of what each
different receiver has or doesn't have.  That's what I used to do
before rsync.

-- 
-
| Phil Howard KA9WGN   | http://linuxhomepage.com/  
http://ham.org/ |
| (first name) at ipal.net | http://phil.ipal.org/   
http://ka9wgn.ham.org/ |
-
-- 
To unsubscribe or change options: 
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: Device majors incorrectly set to 0 during rsync

[Williams, Tom]

Excellent!  In the meantime, the 32bit + -D_FILE_OFFSET_BITS=64 works
brilliantly, so I'll wait for a release.  This is meant to run on prod
boxes, so I need to use a release if possible, so noone asks funny questions
about beta software, etc.

You know how it is.

I just have to remember to compile 32bit until I see a fix go through for
this one.

Thanks for your help :)
Tom Williams


-Original Message-
From: Wayne Davison
To: Williams, Tom
Cc: '[EMAIL PROTECTED]'
Sent: 4/8/2004 2:42 PM
Subject: Re: Device majors incorrectly set to 0 during rsync

On Thu, Apr 08, 2004 at 02:33:29PM -0700, Wayne Davison wrote:
> If you could retry things using the pre-release + patch, I'd
> appreciate it!

Note that my patch won't fix the problem because rsync only transmits
the lower 32 bits of the rdev value.  To fix this will require a
protocol change.  I'll consider that.  Comments welcomed.

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: Device majors incorrectly set to 0 during rsync

[Williams, Tom]

We seem to working from wildly different revs of flist.c :)  Mine is the
stock file from the 2.6.0 source.  I don't even find an occurance of 0xFF in
there, so your patch wouldn't apply, even by hand -- the code is just too
different.

I think I understand what you were attempting to accomplish with it though,
and the relevant equivalent code in the 2.6.0 version /is/ masked as a
DEV64_T 
 if (preserve_devices && IS_DEVICE(file->mode))
file->rdev =
(flags & SAME_RDEV) ? last_rdev : (DEV64_T) read_int(f);

Which /seems/ to do the right thing.

Thanks,
Tom Williams

-Original Message-
From: Wayne Davison
To: Williams, Tom
Cc: '[EMAIL PROTECTED]'
Sent: 4/8/2004 2:25 PM
Subject: Re: Device majors incorrectly set to 0 during rsync

On Thu, Apr 08, 2004 at 01:43:18PM -0700, Williams, Tom wrote:
> Major numbers on 64bit Solaris are being dropped, and created as 0 on
> recieving end.  32bit version works perfectly.

Looks like the code is masking a 64-bit number with a 32-bit mask
(ouch)!  Try the attached patch and let me know if it fixes the problem.

..wayne..
 <> 
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Device majors incorrectly set to 0 during rsync

[Wayne Davison]

On Thu, Apr 08, 2004 at 02:33:29PM -0700, Wayne Davison wrote:
> If you could retry things using the pre-release + patch, I'd
> appreciate it!

Note that my patch won't fix the problem because rsync only transmits
the lower 32 bits of the rdev value.  To fix this will require a
protocol change.  I'll consider that.  Comments welcomed.

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Device majors incorrectly set to 0 during rsync

[Wayne Davison]

On Thu, Apr 08, 2004 at 01:43:18PM -0700, Williams, Tom wrote:
> rsync  version 2.6.0  protocol version 27

I just noticed that you aren't talking about the 2.6.1 pre-release, so
that patch I just sent won't do you any good unless you first download
that:

http://rsync.samba.org/ftp/rsync/preview/rsync-2.6.1pre-1.tar.gz

If you could retry things using the pre-release + patch, I'd appreciate
it!

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Device majors incorrectly set to 0 during rsync

[Wayne Davison]

On Thu, Apr 08, 2004 at 01:43:18PM -0700, Williams, Tom wrote:
> Major numbers on 64bit Solaris are being dropped, and created as 0 on
> recieving end.  32bit version works perfectly.

Looks like the code is masking a 64-bit number with a 32-bit mask
(ouch)!  Try the attached patch and let me know if it fixes the problem.

..wayne..
--- flist.c 1 Apr 2004 18:04:59 -   1.206
+++ flist.c 8 Apr 2004 21:23:16 -
@@ -367,11 +367,11 @@ void send_file_entry(struct file_struct 
} else
rdev = 0;
} else if (IS_DEVICE(mode)) {
-   if ((file->u.rdev & ~0xFF) == rdev_high)
+   if ((file->u.rdev & ~(DEV64_T)0xFF) == rdev_high)
flags |= XMIT_SAME_HIGH_RDEV;
else {
rdev = file->u.rdev;
-   rdev_high = rdev & ~0xFF;
+   rdev_high = rdev & ~(DEV64_T)0xFF;
}
}
}
@@ -594,7 +594,7 @@ void receive_file_entry(struct file_stru
} else if (IS_DEVICE(mode)) {
if (!(flags & XMIT_SAME_HIGH_RDEV)) {
rdev = (DEV64_T)read_int(f);
-   rdev_high = rdev & ~0xFF;
+   rdev_high = rdev & ~(DEV64_T)0xFF;
} else
rdev = rdev_high | (DEV64_T)read_byte(f);
}
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Device majors incorrectly set to 0 during rsync

[Williams, Tom]

OK.  Didn't see anything about this in the archives, other than an old post
from 2000 about a similar problem, so here goes.  Feel free to contact me if
I can provide anything else useful.

Problem:

Major numbers on 64bit Solaris are being dropped, and created as 0 on
recieving end.  32bit version works perfectly.

Environment:

Solaris 8 or Solaris 9 (haven't tried any other 64bit OSes)
SunPRO cc
CFLAGS = -xO2
CFLAGS += -xarch=v9 (turns on 64 bit compile)

Configure:
--
./configure --enable-debug=no

Build:
--
gmake

Run:

./rsync -av /devices/pseudo/ /tmp/foo

Symptoms:
-
[EMAIL PROTECTED] ls -l /devices/pseudo/
total 0
crw-rw-rw-  1 root sys44,  0 Mar 30 19:30 [EMAIL PROTECTED]:arp
crw---  1 root sys11,202 Mar 30 19:48 [EMAIL PROTECTED]:bge
crw---  1 root sys11,  8 Mar 30 19:30 [EMAIL PROTECTED]:eri
crw---  1 root sys11,  7 Mar 30 19:30 [EMAIL PROTECTED]:hme
crw---  1 root sys11, 40 Mar 30 19:30 [EMAIL PROTECTED]:le
crw-rw-rw-  1 root sys11,107 Mar 30 19:35 [EMAIL PROTECTED]:llc1
crw---  1 root sys11,  4 Mar 30 19:35 [EMAIL PROTECTED]:logindmux
crw-rw-rw-  1 root sys11, 23 Mar 30 19:35 [EMAIL PROTECTED]:ptmx
crw--w  1 root tty 0,  0 Apr  8 12:49 [EMAIL PROTECTED]:console
crw--w  1 root tty 0,  0 Apr  7 16:32 [EMAIL PROTECTED]:syscon
crw--w  1 root tty 0,  0 Mar 30 19:30 [EMAIL PROTECTED]:systty
---snip---

[EMAIL PROTECTED] ls -l /tmp/foo/
total 0
crw-rw-rw-  1 root sys   0,  0 Mar 30 19:30 [EMAIL PROTECTED]:arp
crw---  1 root sys   0,202 Mar 30 19:48 [EMAIL PROTECTED]:bge
crw---  1 root sys   0,  8 Mar 30 19:30 [EMAIL PROTECTED]:eri
crw---  1 root sys   0,  7 Mar 30 19:30 [EMAIL PROTECTED]:hme
crw---  1 root sys   0, 40 Mar 30 19:30 [EMAIL PROTECTED]:le
crw-rw-rw-  1 root sys   0,107 Mar 30 19:35 [EMAIL PROTECTED]:llc1
crw---  1 root sys   0,  4 Mar 30 19:35 [EMAIL PROTECTED]:logindmux
crw-rw-rw-  1 root sys   0, 23 Mar 30 19:35 [EMAIL PROTECTED]:ptmx
crw--w  1 root tty   0,  0 Apr  8 12:49 [EMAIL PROTECTED]:console
crw--w  1 root tty   0,  0 Apr  7 16:32 [EMAIL PROTECTED]:syscon
crw--w  1 root tty   0,  0 Mar 30 19:30 [EMAIL PROTECTED]:systty
---snip---

Version/Compile info:
-
[EMAIL PROTECTED] rsync --version
rsync  version 2.6.0  protocol version 27
Copyright (C) 1996-2004 by Andrew Tridgell and others

Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
  no IPv6, 64-bit system inums, 64-bit internal inums


Thanks
Tom Williams
> twilliams
> at
> corio
> dot
> com
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Q: --one-file-system and nested file systems

[Brian McEntire]

Answered my own question... here's what I came up with in case anyone else
wants the answer:


FILESYSTEMS=`ssh -q [EMAIL PROTECTED] df -l|awk '/\// {print $NF}'|tr '\n' ''`

rsync -av -e 'ssh -q' --one-file-system --numeric-ids --relative \
--delete [EMAIL PROTECTED]:"${FILESYSTEMS}" /backups/$SRCHOST


I used tr to translate the carriage return from the df output into a 
space. Then I used double quotes around the $FILESYSTEMS variable on the 
rsync line so that the variable could be interpreted but the value still 
quoted.

If there are any better ways to do this, I'm still all ears.


On Thu, 8 Apr 2004, Brian McEntire wrote:

> Thanks Wayne! 
> 
> > reading of the files on the source machine.  However, you can include
> > all the source filesystems as args in a single copy command and it will
> > enforce the single-filesystem (inode-based) restriction separately for
> > each arg you specify:
> > 
> >   rsync -avxR -e ssh --numeric-ids --delete \
> > --exclude-from=/backups/control/all.exclude \
> > [EMAIL PROTECTED]:'/ /usr /usr/local /usr/local/apache' /backups/B
> 
> That works great from the command line. One follow-up question for anyone 
> with good shell scripting experience:
> 
> I'm trying to automate this with a Bash script to grab file systems from a
> couple of remote hosts.  Here's how I get a list of the file systems local
> to the remote hosts within the script:
> 
> FILESYSTEMS=`ssh $remHost df -l|awk '/\// {print $NF}'`
> 
> Variable FILESYSTEMS then contains, for example:
> /
> /boot
> /var
> 
> But when try to make the rsync call from my script:
>   rsync ... [EMAIL PROTECTED]:\'$FILESYSTEMS\' /backups/B
> 
> I can't get the quoting right. I don't know if its me (probably :-) or the 
> shell or rsync. I've tried variations with '' or " but no luck yet.
> 
> The script contains:
> 
> rsync -av -e 'ssh -q' --one-file-system --numeric-ids --relative \
> --delete --exclude-from=$CONTROL/all.exclude $CONDEXCLUDE \
> [EMAIL PROTECTED]:\'${FILESYSTEMS}\' . 
>  
> But here's what tries to get executed (set -x in the script shows each
> command to be executed):
> 
> rsync -av -e 'ssh -q' --one-file-system --numeric-ids --relative --delete
> --exclude-from=/backups/control/all.exclude '[EMAIL PROTECTED]:'\''/' /boot /dev/shm
> '/sandbox'\''' .
> 
> 
> Does anyone know a good way to quote or set the FILESYSTEMS variable so I 
> can effectively run, from within a script:
>   rsync ... [EMAIL PROTECTED]:$FILESYSTEMS that will equate to [EMAIL PROTECTED]:'/ 
> /boot /var' 
> 
> 
> Thanks a bunch!! 
> 
> Not that it matters, I'm amazed by the patches and responses on this list!
> Its awesome; reminds me of the old days of the Internet.  :-)
> 
> 

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Q: --one-file-system and nested file systems

[Brian McEntire]

Thanks Wayne! 

> reading of the files on the source machine.  However, you can include
> all the source filesystems as args in a single copy command and it will
> enforce the single-filesystem (inode-based) restriction separately for
> each arg you specify:
> 
>   rsync -avxR -e ssh --numeric-ids --delete \
> --exclude-from=/backups/control/all.exclude \
> [EMAIL PROTECTED]:'/ /usr /usr/local /usr/local/apache' /backups/B

That works great from the command line. One follow-up question for anyone 
with good shell scripting experience:

I'm trying to automate this with a Bash script to grab file systems from a
couple of remote hosts.  Here's how I get a list of the file systems local
to the remote hosts within the script:

FILESYSTEMS=`ssh $remHost df -l|awk '/\// {print $NF}'`

Variable FILESYSTEMS then contains, for example:
/
/boot
/var

But when try to make the rsync call from my script:
  rsync ... [EMAIL PROTECTED]:\'$FILESYSTEMS\' /backups/B

I can't get the quoting right. I don't know if its me (probably :-) or the 
shell or rsync. I've tried variations with '' or " but no luck yet.

The script contains:

rsync -av -e 'ssh -q' --one-file-system --numeric-ids --relative \
--delete --exclude-from=$CONTROL/all.exclude $CONDEXCLUDE \
[EMAIL PROTECTED]:\'${FILESYSTEMS}\' . 
 
But here's what tries to get executed (set -x in the script shows each
command to be executed):

rsync -av -e 'ssh -q' --one-file-system --numeric-ids --relative --delete
--exclude-from=/backups/control/all.exclude '[EMAIL PROTECTED]:'\''/' /boot /dev/shm
'/sandbox'\''' .


Does anyone know a good way to quote or set the FILESYSTEMS variable so I 
can effectively run, from within a script:
  rsync ... [EMAIL PROTECTED]:$FILESYSTEMS that will equate to [EMAIL PROTECTED]:'/ 
/boot /var' 


Thanks a bunch!! 

Not that it matters, I'm amazed by the patches and responses on this list!
Its awesome; reminds me of the old days of the Internet.  :-)

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [librsync-devel] librsync and rsync vulnerability to maliciously crafted data. was Re: MD4 checksum_seed

[Wayne Davison]

On Thu, Apr 08, 2004 at 03:50:48PM +1000, Donovan Baarda wrote:
> I think I've just realised what you were getting at; if the
> checksum_seed is based on something like the whole file md4sum, it
> becomes repeatable, but unpredictable.

Not so.  Copy the file once, and you'd get all the data you'd need to
create a new local file using a known-signature attack (as long as the
input file didn't change, and that's easy to predict for many files).
I also don't like the doubling of the I/O-cost on the sending side, so
I don't think this is a good way to go.

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync is great

[Brian Cuttler]

Use # find on the remote system and list | rm files that meet
your date criteria ?

> I'm not on the list but I just wanted to throw my positive feedback out
> there.  I've been using rsync to backup work that I generate within a
> lab to an offsite system.  If I lost this data ...
> 
> So great.
> 
> 
> In standard community fashion, though, I want to throw an idea out
> there.  It would be nice if there were an option like:
> 
>   --delete-if-too-old
> 
> 
> Currently I run a cron job which copies files offsite nightly.  Once a
> month, though, I use the --delete option to get rid of cruft.
> 
> It would be _nice_, though, if I could delete things that were older
> than a month instead of a simple comparison between what's on the sender
> and the receiver.
> 
> This way files would have a known age of 30 days or whatever before they
> get tossed.  As it is, the interval I can regress to has to do with the 
> time of the month, which is just too close to real life ...
> 
> 
> If there is a better way to submit suggestions I will do so upon hearing
> about it.  
> 
> 
> Thanks for rsync.  I use it every day.
> -- 
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [librsync-devel] librsync and rsync vulnerability to maliciously crafted data. was Re: MD4 checksum_seed

[Eran Tromer]

Ahoy,

On 2004/04/08 14:16, Donovan Baarda wrote:
>>Nice indeed, but the cost is enormous: you'll have to read the file
>>twice. When syncing a mostly-unchanged file that's larger than the disk
>>cache, that means doubling the runtime (and disk load) on the receiver's
>>side. Also, it means 'rdiff signature' and equivalents won't work on
> 
> But the vast majority of the load is in the delta calculation on the
> sender's side.

My experience is that when you sync a mostly unchanged large files on
modern PCs, both sides are IO-bound. The delta calculation just rolls
along at top speed due to the "try the next block first" heuristic.


>>I'm afraid it's still vulnerable to case 3 (a pair of "target" and
>>"original" files with matching blocks). For simplicity consider
>>single-block files. In this case what you've done is simply to replace
>>the hash function
>>  f(x) = truncate(MD4(x,fixed_seed))
>>with the hash function
>>  f'(x) = truncate(MD4(x,MD4(x,fixed_seed)))
> 
> Not quite... it's f(x,y) = truncate(MD4(x,MD4(y,fixed_seed))), where x and y
> are the two blocks to be compared. This means you have to re-calculate the
> hash for every compare, not just once for every block.

Indeed, you're right.
More fundamentally, every time you compute f(x,y) you win iff
f(x,y)==f(y,y), otherwise you don't learn anything interesting. So
you'll have to compute f about 2^n times. Yes, this looks secure when
the hash function is perfectly random. The only reservation is that
using the same user-affected seed to hash many user-determined blocks is
uncomfortably reminiscent of MD4's known weaknesses.

Still, are there reasons beyond the aesthetic to want deterministic
signature generation? The costs in IO and flexibility seem very high.

  Eran

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [librsync-devel] librsync and rsync vulnerability to maliciously crafted data. was Re: MD4 checksum_seed

[Donovan Baarda]

G'day again,

From: "Eran Tromer" <[EMAIL PROTECTED]>
[...]
> > if the
> > checksum_seed is based on something like the whole file md4sum, it
> > becomes repeatable, but unpredictable. You can't manipulate individual
> > blocks without it affecting every other blocksum, but the signature for
> > the same file is always the same. Nice :-)
>
> Nice indeed, but the cost is enormous: you'll have to read the file
> twice. When syncing a mostly-unchanged file that's larger than the disk
> cache, that means doubling the runtime (and disk load) on the receiver's
> side. Also, it means 'rdiff signature' and equivalents won't work on

But the vast majority of the load is in the delta calculation on the
sender's side.

> arbitrary streams. Currently the receiver can tee the output of 'rdiff
> patch' into both the output file and an instance of 'rdiff signature',
> in order to save the IO of re-reading the file upon the next sync; this
> won't work anymore. (How about a built-in option for that, BTW?).

True... that is a nice feature... you are slowly turning me off the
whole-file checksum as seed :-)

> > More thoughts on using the wholefile sum as the seed; at first I thought
> > this would still be vulnerable to case 3). Using a any single block as
> > the original file and trying to find any block that matches means you
> > still have "birthday algorithm" numbers of blocks to check (ie 2^(n/2)).
> > However, each block "comparison" requires the recalculation of the
> > "target" blocksum using the "original" blocksum as the seed, resulting
> > in non-birthday algorithm number of checksum calculations (ie, 2^n).
>
> I'm afraid it's still vulnerable to case 3 (a pair of "target" and
> "original" files with matching blocks). For simplicity consider
> single-block files. In this case what you've done is simply to replace
> the hash function
>   f(x) = truncate(MD4(x,fixed_seed))
> with the hash function
>   f'(x) = truncate(MD4(x,MD4(x,fixed_seed)))

Not quite... it's f(x,y) = truncate(MD4(x,MD4(y,fixed_seed))), where x and y
are the two blocks to be compared. This means you have to re-calculate the
hash for every compare, not just once for every block.

> which happens to be the same as hashing two copies of x in sequence.
> But the birthday attack doesn't care what's the hash function; it only
> cares about its input and output sizes (which we didn't change) and that
> the function is "sufficiently random".

It also assumes that the hash is done once per sample input, not once per
compare. Sure, you still only need to try 2^(n/2) blocks, but you need to
calculate 2^n hashes, and that's what really hurts.


Donovan Baardahttp://minkirri.apana.org.au/~abo/




-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: [librsync-devel] librsync and rsync vulnerability to maliciously crafted data. was Re: MD4 checksum_seed

[Eran Tromer]

On 2004/04/08 08:50, Donovan Baarda wrote:
>>In some cases you might prefer to actually store an signed signature
>>using something like GPG.

I think librsync should act as a black box that guarantees file
integrity (which, apparently, requires a whole file checksum). If
someone wants to add authentication or encryption or whatever on top of
that, all the merrier, but let that be done in addition to rsync's own
integrity check.

That means both librsync and (say) GPG would compute a whole file hash.
The communication cost of this duplication is negligible (a typical hash
is much smaller than a typical digital signature). The computation is a
bit more annoying -- it will roughly double the CPU consumption of the
receiver. That can be solved that by revealing the whole file checksum
(when available) via the API, so that librsync users can directly sign
that checksum instead of computing their own.


> if the
> checksum_seed is based on something like the whole file md4sum, it
> becomes repeatable, but unpredictable. You can't manipulate individual
> blocks without it affecting every other blocksum, but the signature for
> the same file is always the same. Nice :-)

Nice indeed, but the cost is enormous: you'll have to read the file
twice. When syncing a mostly-unchanged file that's larger than the disk
cache, that means doubling the runtime (and disk load) on the receiver's
side. Also, it means 'rdiff signature' and equivalents won't work on
arbitrary streams. Currently the receiver can tee the output of 'rdiff
patch' into both the output file and an instance of 'rdiff signature',
in order to save the IO of re-reading the file upon the next sync; this
won't work anymore. (How about a built-in option for that, BTW?).


> More thoughts on using the wholefile sum as the seed; at first I thought
> this would still be vulnerable to case 3). Using a any single block as
> the original file and trying to find any block that matches means you
> still have "birthday algorithm" numbers of blocks to check (ie 2^(n/2)).
> However, each block "comparison" requires the recalculation of the
> "target" blocksum using the "original" blocksum as the seed, resulting
> in non-birthday algorithm number of checksum calculations (ie, 2^n).

I'm afraid it's still vulnerable to case 3 (a pair of "target" and
"original" files with matching blocks). For simplicity consider
single-block files. In this case what you've done is simply to replace
the hash function
  f(x) = truncate(MD4(x,fixed_seed))
with the hash function
  f'(x) = truncate(MD4(x,MD4(x,fixed_seed)))
which happens to be the same as hashing two copies of x in sequence.
But the birthday attack doesn't care what's the hash function; it only
cares about its input and output sizes (which we didn't change) and that
the function is "sufficiently random".

  Eran

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: librsync and rsync vulnerability to maliciously crafted data. was Re: MD4 checksum_seed

[Donovan Baarda]

G'day,

From: "Eran Tromer" <[EMAIL PROTECTED]>
[...]
> > librsync needs a whole file checksum. Without it, it silently fails for
> > case 1), 3), and 4).
> >
> > librsync could benefit from a random checksum_seed. It would need to be
> > included in the signature. Without it librsync is vulnerable to cases 1)
> > and 3).
> [snip]
> > rsync shouldn't need a fixed seed for batch modes... just store the seed
> > in the signature. using a fixed seed makes it vulnerable to 1) and 3).
>
> I fully agree with your analysis.
> I'll just note that in many situations, case 2 can be elevated to case 3
> simply by transferring the file twice.

Yeah... did you see my followup post about the posiblity of using the
whole-file checksum as the checksum_seed for the blocksums? I think it would
be a good idea for librsync. It does require a double-parse to generate the
signature, but is otherwise quite nice.


Donovan Baardahttp://minkirri.apana.org.au/~abo/




-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: librsync and rsync vulnerability to maliciously crafted data. was Re: MD4 checksum_seed

[Eran Tromer]

Hi,

On 2004/04/05 07:21, Donovan Baarda wrote:
[snip]
> there are four ways crafted blocks can be use;
> 
> 1) two crafted blocks in the "original" file
> 
> 2) two crafted blocks in the "target" file
> 
> 3) a crafted pair of "target" and "original" files with matching
> block(s)
> 
> 4) a block in the "target" crafted to match a block in the "original"
[snip]
> Summary;
> 
> case 2) has no impact
> 
> case 4) is of minimal impact in rsync, and sufficiently hard in
> librsync.
> 
> librsync needs a whole file checksum. Without it, it silently fails for
> case 1), 3), and 4).
> 
> librsync could benefit from a random checksum_seed. It would need to be
> included in the signature. Without it librsync is vulnerable to cases 1)
> and 3).
[snip]
> rsync shouldn't need a fixed seed for batch modes... just store the seed
> in the signature. using a fixed seed makes it vulnerable to 1) and 3).

I fully agree with your analysis.
I'll just note that in many situations, case 2 can be elevated to case 3
simply by transferring the file twice.

  Eran
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync is great

[Eric Ortega]

I'm not on the list but I just wanted to throw my positive feedback out
there.  I've been using rsync to backup work that I generate within a
lab to an offsite system.  If I lost this data ...

So great.


In standard community fashion, though, I want to throw an idea out
there.  It would be nice if there were an option like:

  --delete-if-too-old


Currently I run a cron job which copies files offsite nightly.  Once a
month, though, I use the --delete option to get rid of cruft.

It would be _nice_, though, if I could delete things that were older
than a month instead of a simple comparison between what's on the sender
and the receiver.

This way files would have a known age of 30 days or whatever before they
get tossed.  As it is, the interval I can regress to has to do with the 
time of the month, which is just too close to real life ...


If there is a better way to submit suggestions I will do so upon hearing
about it.  


Thanks for rsync.  I use it every day.
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --suffix problem - possibly bug?

[Wayne Davison]

On Wed, Apr 07, 2004 at 11:24:46PM -0400, Jim Salter wrote:
> So, just to clarify - this is a bug that needs fixing only on the
> server side, not the client side, correct?

Yes, that's correct.

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html