https://bugzilla.samba.org/show_bug.cgi?id=13735
Bug ID: 13735 Summary: Synchronize files when the sending side has newer change times while modification times and sizes are identical on both sides Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: way...@samba.org Reporter: sbehu...@gmail.com QA Contact: rsync...@samba.org Files that have identical sizes and change times on the sending and receiving sides, but different contents, will not be synchronized by default (e.g. rsync -a --delete source:/path/ dest:/path/). Synchronizing these files requires the use of --checksum or --ignore-times options, which are both sub-optimal in most cases (see caveats below). I would like to propose new options to efficiently synchronize these files. To make this report as clear as possible, I kindly remind that modification times (mtime) can be manually set by users (e.g. with touch) and rsync will preserve those during synchronization (with -a, which includes -t). By contrast, change times (ctime) are automatically updated by the OS when files are changed. rsync relies on modification times to decide whether it should skip or transfer files. There are some use cases where files are modified while preserving their original sizes and mtimes. This can happen when a fixed-size file is updated with new content in a build chain while forcibly preserving a specific mtime. To the best of my knowledge, rsync does not have any option to transfer these files in an efficient manner. I illustrate the issue below: # Create two different files with same size echo 'new content' > srcfile echo 'old content' > destfile # Set identical mtime for both files: Update srcfile's mtime to match destfile's mtime touch -mr destfile srcfile # At this point, srcfile and destfile have: # - identical size # - identical mtime # - different content # - srcfile's ctime is newer than destfile's ctime # rsync experiments (dry runs) rsync -avn srcfile destfile # will not synchronize rsync -avn --checksum srcfile destfile # will synchronize rsync -avn --ignore-times srcfile destfile # will synchronize The desired behavior is to synchronize srcfile to destfile, because srcfile is different and has a newer ctime. Caveats: --checksum option is I/O intensive and will check all files --ignore-times option will force a (re)synchronization of all files To solve this issue, we could check change times. Considering that rsync will not be able to control change times on the receiving side, we must be careful. If we suppose that rsync used ctimes (and not mtimes) to compare these files, it would first find that srcfile is newer, and then on subsequent rsync passes, it would find that destfile is newer. Therefore we can't transfer files if [srcfile's ctime != destfile's ctime]. However we can use a combination of mtime and ctime to solve the above ambiguity. Considering that both files have identical size and mtime, I propose the following new flags: --ignore-times-if-newer "don't skip files that match size and (m)time if source has newer ctime" This would force a transfer for files that are newer on the sending side, regardless of their sizes and mtimes. --checksum-if-newer "skip based on checksum if source has newer ctime" Likewise, this would force a transfer for files that are newer on the sending side and different in content. Similarly, --ignore-times-if-older and --checksum-if-older may be desirable when we trust older files on the sending side more than newer files on the receiving side. New rsync experiments: # Try to synchronize srcfile to destfile rsync -avn --ignore-times-if-newer srcfile destfile # will synchronize rsync -avn --ignore-times-if-older srcfile destfile # will not synchronize # Swap source and destination: Try to synchronize destfile to srcfile rsync -avn --ignore-times-if-newer destfile srcfile # will not synchronize rsync -avn --ignore-times-if-older destfile srcfile # will synchronize # Real examples (wet runs) to test multiple rsync passes # Synchronizing srcfile to destfile rsync -av --ignore-times-if-newer srcfile destfile # 1st pass: will synchronize rsync -av --ignore-times-if-newer srcfile destfile # 2nd pass: will skip rsync -av --ignore-times-if-newer srcfile destfile # nth pass: will always skip: srcfile's ctime is no longer newer, rely only on size and mtime that are identical rsync -av --ignore-times-if-older srcfile destfile # 1st pass: will synchronize rsync -av --ignore-times-if-older srcfile destfile # 2nd pass: will synchronize rsync -av --ignore-times-if-older srcfile destfile # nth pass: will always synchronize: srcfile's ctime will always be older at this point # Final cleanup rm srcfile destfile And similarly for --checksum-if-newer/--checksum-if-older. These additional options would efficiently synchronize newer (--ignore-times-if-newer/--checksum-if-newer) or older (--ignore-times-if-older/--checksum-if-older) files that do not differ in size and mtime. -- You are receiving this mail because: You are the QA Contact for the bug. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html