I think it's very hard to be sure what's going on with --remove-source-files ; I think you should drop that option, look for whether the problem continues, and if you need the files to be cleaned up, do so in a separate step.
In particular as someone else suggested, are you *sure* the original copies of the partial files are actually the size you think they are? I don't ever recall seeing a case where rsync just failed to transfer a file and thought it had succeeded; there pretty much has to be something else going on. On Sat, Mar 04, 2023 at 12:39:52AM -0600, Albert Croft via rsync wrote: > At $work I have an odd situation involving incomplete file transfers, but I > am unsure where the issue may be occurring. Here is the scenario. > > Problem: > Sometimes the file transfer seems to have completed, but the file size does > not match that on the remote system. > > > Details: > I transfer a number of large (>1GB) Tar-Gzipped (.tgz) files via SSH tunnels > from $customer. Because of some previous issues, sometimes the SSH tunnels > may be terminated externally. As a result, I am currently using the 'split' > command to break the files into 1-GB "chunks" (ex.: foo.tgz.aa, foo.tgz.ab, > ...). > > For the rsync transfer, I am using the following options: > rsync -az \ > -e "ssh ..." \ > --link-dest=/local/path1 \ > --link-dest=/local/path2 \ > --remove-source-files \ > user@remote:/path/to/files \ > /local/path1/ > > where > '-e "ssh ..."' is the set of SSH options (for tunneling, etc.). > '--link-dest=/local/path1' refers to a local directory that might > contain a > copy of the file. > '--link-dest=/local/path2' refers to a local directory that might > contain a > copy of the file. > > I am frequently encountering times where the file appears to have been > transferred but is incomplete. (Example: foo.tgz.ab now exists on the local > system, has been removed from the remote, but is incomplete.) > > > Additional notes: > To my knowledge I do not know if the 'gzip' '--rsyncable' option is being > used (but I do not think so--I suspect the file is created using a command > similar to 'tar czf foo.tgz ...'). > > The rsync commands may be launched from command-line or cron, but use the > same format and options in either case. As a result, there may be multiple > rsync processes pulling files from the same remote path to the same local > path. > > I know that when rsync transfers a file (ex.: foo.tgz.ab) that during the > transfer process it is named '.foo.tgz.ab.??????' (where '.??????' is a > 6-character unique extension), and that upon completion the file is renamed > to 'foo.tgz.ab'. (So I may see .foo.tgz.ab.4e67d0 and .foo.tgz.ab.fa7325 in > the directory while the transfers are going.) > > > I am unsure if this is a result of the combination of options I am using, or > where to begin troubleshooting. Any guidance or direction would be > appreciated. > > -Albert C. > > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html