Re: why are these files being transfered
On Sun, Jun 01, 2003 at 01:29:32AM -0700, Harry Putnam wrote: jw schultz [EMAIL PROTECTED] writes: Hmm, -a == -rlptgoD could it be that the timestamp, owner, group or perms differ? No, thats not it (at least not all of it) I see the same behavior with rsync -rCv. Only thing that comes close is `-rCcv'. Even that transfers a few files that diff sees no difference in. Read the section describing --times again. I'd suggest you look again at the files that -c found as differing and diff did not. Is there a major difference between differences as reported by diff and what rsync does? If we ignore timestamp, permissions and ownership. And diff finds no difference between two files. And I even went so far as to run md5sum on a couple. With all that same ness, why would rsync mess with it? Seems one should not need the -c flag if diff reports no difference, and the rsync commmand line is -r only. Rsync does not compare contents before transfer unless -c is used. Without -c meta-data is the only means rsync uses to determine what files have changed. But look at this: reader $ ls -l buf.t/UPDATE base.t/UPDATE -rw-rw-r-- 1 reader reader 2589 May 30 18:06 base.t/UPDATE -rw-r--r-- 1 reader reader 2589 May 29 14:21 buf.t/UPDATE reader $ diff buf.t/UPDATE base.t/UPDATE nothing md5sum buf.t/UPDATE base.t/UPDATE 050a08b8f8fc8b1debdcd387f9bdcc67 buf.t/UPDATE 050a08b8f8fc8b1debdcd387f9bdcc67 base.t/UPDATE rsync -nrCv buf.t/ base.t|wc -l 281 Running with -n so nothing actually happens. And 281 files are processed rsync -nrCv buf.t/ base.t|grep UPDATE UPDATE including UPDATE... Why? Throwing in the -c (checksum checking) and it goes away. Since I've already dropped timestamp owner group access etc it seems rsync should leave this alone without the -c flag. You dropped timestamp preservation but you didn't tell it to ignore timestamps. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: why are these files being transfered
On Fri, May 30, 2003 at 08:02:11PM -0700, Harry Putnam wrote: Why are they being transferred? Undoubtedly because their timestamps are different. Only files that have identical size and timestamp are considered to be identical in an rsync transfer (unless you use -c). In the case of having identical files with different timestamps, rsync will transfer just the checksum data to determine that the files are indeed identical, and move into place a file with an identical timestamp (which looks just like the files are being transferred, but no actual file-data gets sent). ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: why are these files being transfered
On Sat, May 31, 2003 at 01:04:40AM -0700, Harry Putnam wrote: I tested it on a real cvs checked out module and just as I suspected after rsyncing from the buffer directory where diff reports no differnces. Then try cvs commit. Cvs demands every file be commited. If you are syncing two cvs-checked-out areas, you must make sure NOT to use the -C (--cvs-exclude) option or you won't update the files in the CVS dir, and that will cause you problems. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: why are these files being transfered
Wayne Davison [EMAIL PROTECTED] writes: On Sat, May 31, 2003 at 01:04:40AM -0700, Harry Putnam wrote: I tested it on a real cvs checked out module and just as I suspected after rsyncing from the buffer directory where diff reports no differnces. Then try cvs commit. Cvs demands every file be commited. If you are syncing two cvs-checked-out areas, you must make sure NOT to use the -C (--cvs-exclude) option or you won't update the files in the CVS dir, and that will cause you problems. No.. one is not a checked out module ... the source of rsync command is nearly a mirror but no CVS directories Wayne Davison [EMAIL PROTECTED] writes: On Fri, May 30, 2003 at 08:02:11PM -0700, Harry Putnam wrote: Why are they being transferred? Undoubtedly because their timestamps are different. Only files that have identical size and timestamp are considered to be identical in an rsync transfer (unless you use -c). In the case of having identical files with different timestamps, rsync will transfer just the checksum data to determine that the files are indeed identical, and move into place a file with an identical timestamp (which looks just like the files are being transferred, but no actual file-data gets sent). But it seem cvs then sees them as modified. Not sure I see the point of any modification, time stamp or whatever if the files are the same. Further, I've blown one of my test apparently. I thought I saw the same behavior with --size-only flag set but trying now a few times I see --size-only does cure the problem. Maybe I'm going at this backward. Would it satisfy cvs if I followed this proceedure: Here /new/base.t is the checked out module ./cvs_buf.t is a near mirror but with no CVS directories. Revisions happen here and are then to be rsynced to the checked out module. rsync -navvzC --size-only ./cvs_buf.t/ /new/base.t Once this transfer is done and any changes in /cvs_buf.t are transferred to /new/base run cvs commit Now somehow transfer only timestamps from /new/base.t to ./cvs_buf.t I thought I remembered a flag for that purpose, but don't see it now in man rsync. Maybe I'm thinking of ftp or something. A further questions here: A section of man page gives a rough sketch of something a lot like what I want to do: To synchronize my samba source trees I use the following Makefile targets: get: rsync -avuzb --exclude '*~' samba:samba/ . put: rsync -Cavuzb . samba:samba/ sync: get put Do you know the details of how this works? What is meant by `Makefile targets'. and what is happening above. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: why are these files being transfered
On Sat, May 31, 2003 at 09:24:19AM -0700, Harry Putnam wrote: But it seem cvs then sees them as modified. Them who? You said you're transferring the files from the cvs checkout area to a buffer area. Are you transferring them back again? Not sure I see the point of any modification, time stamp or whatever if the files are the same. But, how do you know they're the same? Rsync certainly doesn't (though, if you use my suggested -c option, it would determine that for itself). The question should be, how are things getting out-of-whack between the two areas? You should be transferring the data with -a, and then it should just work. It's not clear what you're trying to do, but I assume there's a flaw in your use of rsync. Either that, or you're not aware that 2 different CVS checkouts of the same CVS module will have radically different modification times on all the files even when they have identical content. I see --size-only does cure the problem. That would be an extraordinarily bad idea. The only thing that --size-only should be used with are files that can't possibly change unless their size changes (such as log files). Since source files can change and remain the same size, so this is not a solution. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: why are these files being transfered
Wayne Davison [EMAIL PROTECTED] writes: On Sat, May 31, 2003 at 09:24:19AM -0700, Harry Putnam wrote: But it seem cvs then sees them as modified. Them who? You said you're transferring the files from the cvs checkout area to a buffer area. Are you transferring them back again? Them ... files in base.t. No its the other way round. And only one direction OK, let me try to be more clear. I thought this; from my original post was good: The setup employs a bufffer where changes are made then moved to a checked out module from cvs repository and finally any cvs actions necessary are done. Since my test used a directory named cvs_buf.t I thought it would be clear it was the buffer area mentioned above. So here is what I have in better detail: TARGET of rsync command: ./base.t == actual checked out cvs module SOURCE of rsync command ./cvs_buf.t== an area that is an approximate mirror of cvs files in ./base.t Changes happen here but most (nearly all) files are identical to their cousins in `base.t'. (except possibly timestamp The idea is to do actual modificatons edit of files etc in ./cvs_buf.t Then rsync cvs_buf.t to base.t base.t contains the `CVS' directories since it is a checked out module. All cvs commands will be done in ./base.t so that will take care of the `CVS' directorires. They are not involved in the rsync action. Harry wrote: Not sure I see the point of any modification, time stamp or whatever if the files are the same. [...] Wayne answered: But, how do you know they're the same? Rsync certainly doesn't (though, if you use my suggested -c option, it would determine that for itself). [...] Well the posted results of the diff commands found no difference however rsync transfres or modifies the file in target anyway. rsync transfer (unless you use -c). In the case of having identical files with different timestamps, rsync will transfer just the checksum data to determine that the files are indeed identical, and move into place a file with an identical timestamp (which looks just like the files are being transferred, but no actual file-data gets sent). Looks like the -c flag is what I need alright. But I still can't understand why rsync would modifie a file where only the time stamp is different. As you say it `moves into place a file with identical timestamp. But no data is transfered. However cvs tools do think they are modified. Please observe this series of tests that display what is confusing me. First a fresh `cvs co' of module base.t cvs co base.t [...] snipped output We now have ./base.t Again ./cvs_buf.t Is a tree of files that is a replical of ./base.t However timestamps may be different. diff base.t cvs_buf.t Only in base.t: CVS Common subdirectories: base.t/common and cvs_buf.t/common Common subdirectories: base.t/lgate and cvs_buf.t/lgate Common subdirectories: base.t/obsd and cvs_buf.t/obsd Common subdirectories: base.t/reader and cvs_buf.t/reader Common subdirectories: base.t/sol and cvs_buf.t/sol diff finds no difference except the presence CVS in base.t Now diffing a specific file: diff base.t/reader/etc/fstab cvs_buf.t/reader/etc/fstab Again no difference is found. Now we run the rsync command: (not using -c here) rsync -avvzC cvs_buf.t/ base.t 21| tee /root/cvs_buf.test (note the -C is being used for its other propertied not to ignore CVS since there are no CVS dir in cvs_buf.t) 289 lines of output including reader/etc/fstab [...] and ending wrote 641047 bytes read 5560 bytes 431071.33 bytes/sec total size is 1738494 speedup is 2.69 Now lets see what cvs thinks of it: cd base.t cvs commit Gives me a vi buffer where every last file is listed as needing commit. So apparently, at least in the eyes of cvs, every last file has been changed. Since files were the same to start with, this seems wrong. I do see where -c will cure it, but I'm asking why does rsync do that when the files are the same? The question should be, how are things getting out-of-whack between the two areas? You should be transferring the data with -a, All my posted rsync commands used the -a flag. and then it should just work. It's not clear what you're trying to do, but I assume there's a flaw in your use of rsync. Either that, or you're not aware that 2 different CVS checkouts of the same CVS module will have radically different modification times on all the files even when they have identical content. Now with a clearer picture of what I'm doing. And if I haven't worn out your kindness with all the included data, do you see what I'm doing wrong? And is it appropriate for rsync to alter files that have nothing differnt in them? I see --size-only does cure the problem. That would be an extraordinarily bad idea. The only thing that --size-only should be used
Re: why are these files being transfered
jw schultz [EMAIL PROTECTED] writes: Hmm, -a == -rlptgoD could it be that the timestamp, owner, group or perms differ? No, thats not it (at least not all of it) I see the same behavior with rsync -rCv. Only thing that comes close is `-rCcv'. Even that transfers a few files that diff sees no difference in. Is there a major difference between differences as reported by diff and what rsync does? If we ignore timestamp, permissions and ownership. And diff finds no difference between two files. And I even went so far as to run md5sum on a couple. With all that same ness, why would rsync mess with it? Seems one should not need the -c flag if diff reports no difference, and the rsync commmand line is -r only. But look at this: reader $ ls -l buf.t/UPDATE base.t/UPDATE -rw-rw-r-- 1 reader reader 2589 May 30 18:06 base.t/UPDATE -rw-r--r-- 1 reader reader 2589 May 29 14:21 buf.t/UPDATE reader $ diff buf.t/UPDATE base.t/UPDATE nothing md5sum buf.t/UPDATE base.t/UPDATE 050a08b8f8fc8b1debdcd387f9bdcc67 buf.t/UPDATE 050a08b8f8fc8b1debdcd387f9bdcc67 base.t/UPDATE rsync -nrCv buf.t/ base.t|wc -l 281 Running with -n so nothing actually happens. And 281 files are processed rsync -nrCv buf.t/ base.t|grep UPDATE UPDATE including UPDATE... Why? Throwing in the -c (checksum checking) and it goes away. Since I've already dropped timestamp owner group access etc it seems rsync should leave this alone without the -c flag. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: why are these files being transfered
Harry Putnam [EMAIL PROTECTED] writes: No, thats not it (at least not all of it) I see the same behavior with rsync -rCv. Only thing that comes close is `-rCcv'. Even that transfers a few files that diff sees no difference in. CORRECTION ALERT That last line: Even that transfers a few files that diff sees no difference in. is incorrect. NO such test results were obtained. However the first part stands and was tested several times. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: why are these files being transfered
Harry Putnam [EMAIL PROTECTED] writes: But when I run rsync it appear to want to transfer massive numbers of files. rsync -avvzC /root/cvs_buf.t/ /root/base.t 21| tee /root/cvs_buf.test [...] sol/etc/SYS_ENV [...] I tested it on a real cvs checked out module and just as I suspected after rsyncing from the buffer directory where diff reports no differnces. Then try cvs commit. Cvs demands every file be commited. Surely this is not what is supposed to happen. rsync man pages indicate rsync will work nicely with cvs. Further with rsyncs elaborate algorythems for testing for differences why are I seeing the massive transfer when files are the same? -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: why are these files being transfered
On Sat, May 31, 2003 at 01:04:40AM -0700, Harry Putnam wrote: Harry Putnam [EMAIL PROTECTED] writes: But when I run rsync it appear to want to transfer massive numbers of files. rsync -avvzC /root/cvs_buf.t/ /root/base.t 21| tee /root/cvs_buf.test [...] sol/etc/SYS_ENV [...] I tested it on a real cvs checked out module and just as I suspected after rsyncing from the buffer directory where diff reports no differnces. Then try cvs commit. Cvs demands every file be commited. Surely this is not what is supposed to happen. rsync man pages indicate rsync will work nicely with cvs. Further with rsyncs elaborate algorythems for testing for differences why are I seeing the massive transfer when files are the same? Hmm, -a == -rlptgoD could it be that the timestamp, owner, group or perms differ? -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
