Re: Trying to diagnose incomplete file transfer
Hi On Sun, 05 Mar 2023 07:45:16 -0800 Robin Lee Powell via rsync wrote: > Oh, yeah, I missed that part. Yeah, don't do that; it's easy to add > a lock file to a shell script. Not so easy IMO to do that properly. Use the flock command if your system provides it. That said, using a systemd service (resp timer) for interactive (resp batch/cron) use: - gives you that locking for free - enforce using the same execution context in both cases (ex: the Unix environment, the locale ...) -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trying to diagnose incomplete file transfer
Oh, yeah, I missed that part. Yeah, don't do that; it's easy to add a lock file to a shell script. On Sun, Mar 05, 2023 at 12:30:16PM +0100, Hardy via rsync wrote: > I second Francis here. You don't need to diagnose incomplete file transfers > as long as you have racing conditions as you described. This leads to strange > result inevitably. > NEVER start several rsync jobs manipulating the same data - especially if > there are modifications to BOTH sides source and destination. > > You do not necessarily define a service like Francis suggests. A simple > semaphore approach suffices. Perhaps even something like > > # ps fax | grep -v grep | grep $0 && exit > > to prevent this exact command "$0" to start concurrently. > > Hardy > > Am 04.03.23 um 08:38 schrieb Francis.Montagnac--- via rsync: > > > > Hi. > > > > On Sat, 04 Mar 2023 00:39:52 -0600 Albert Croft via rsync wrote: > > > > > The rsync commands may be launched from command-line or cron, but use > > > the same format and options in either case. As a result, there may be > > > multiple rsync processes pulling files from the same remote path to the > > > same local path. > > > > I think you should first prevent this to happen. > > > > If your receiving machine is using systemd: > > > >- define a X.service for doing the rsync > >- define a X.timer unit to replace using cron > >- launch from the command line with: systemctl start X > > - this will not start a new rsync if one runs already (if X.service is > >running) > > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trying to diagnose incomplete file transfer
I second Francis here. You don't need to diagnose incomplete file transfers as long as you have racing conditions as you described. This leads to strange result inevitably. NEVER start several rsync jobs manipulating the same data - especially if there are modifications to BOTH sides source and destination. You do not necessarily define a service like Francis suggests. A simple semaphore approach suffices. Perhaps even something like # ps fax | grep -v grep | grep $0 && exit to prevent this exact command "$0" to start concurrently. Hardy Am 04.03.23 um 08:38 schrieb Francis.Montagnac--- via rsync: Hi. On Sat, 04 Mar 2023 00:39:52 -0600 Albert Croft via rsync wrote: The rsync commands may be launched from command-line or cron, but use the same format and options in either case. As a result, there may be multiple rsync processes pulling files from the same remote path to the same local path. I think you should first prevent this to happen. If your receiving machine is using systemd: - define a X.service for doing the rsync - define a X.timer unit to replace using cron - launch from the command line with: systemctl start X - this will not start a new rsync if one runs already (if X.service is running) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trying to diagnose incomplete file transfer
I think it's very hard to be sure what's going on with --remove-source-files ; I think you should drop that option, look for whether the problem continues, and if you need the files to be cleaned up, do so in a separate step. In particular as someone else suggested, are you *sure* the original copies of the partial files are actually the size you think they are? I don't ever recall seeing a case where rsync just failed to transfer a file and thought it had succeeded; there pretty much has to be something else going on. On Sat, Mar 04, 2023 at 12:39:52AM -0600, Albert Croft via rsync wrote: > At $work I have an odd situation involving incomplete file transfers, but I > am unsure where the issue may be occurring. Here is the scenario. > > Problem: > Sometimes the file transfer seems to have completed, but the file size does > not match that on the remote system. > > > Details: > I transfer a number of large (>1GB) Tar-Gzipped (.tgz) files via SSH tunnels > from $customer. Because of some previous issues, sometimes the SSH tunnels > may be terminated externally. As a result, I am currently using the 'split' > command to break the files into 1-GB "chunks" (ex.: foo.tgz.aa, foo.tgz.ab, > ...). > > For the rsync transfer, I am using the following options: > rsync -az \ > -e "ssh ..." \ > --link-dest=/local/path1 \ > --link-dest=/local/path2 \ > --remove-source-files \ > user@remote:/path/to/files \ > /local/path1/ > > where > '-e "ssh ..."' is the set of SSH options (for tunneling, etc.). > '--link-dest=/local/path1' refers to a local directory that might > contain a > copy of the file. > '--link-dest=/local/path2' refers to a local directory that might > contain a > copy of the file. > > I am frequently encountering times where the file appears to have been > transferred but is incomplete. (Example: foo.tgz.ab now exists on the local > system, has been removed from the remote, but is incomplete.) > > > Additional notes: > To my knowledge I do not know if the 'gzip' '--rsyncable' option is being > used (but I do not think so--I suspect the file is created using a command > similar to 'tar czf foo.tgz ...'). > > The rsync commands may be launched from command-line or cron, but use the > same format and options in either case. As a result, there may be multiple > rsync processes pulling files from the same remote path to the same local > path. > > I know that when rsync transfers a file (ex.: foo.tgz.ab) that during the > transfer process it is named '.foo.tgz.ab.??' (where '.??' is a > 6-character unique extension), and that upon completion the file is renamed > to 'foo.tgz.ab'. (So I may see .foo.tgz.ab.4e67d0 and .foo.tgz.ab.fa7325 in > the directory while the transfers are going.) > > > I am unsure if this is a result of the combination of options I am using, or > where to begin troubleshooting. Any guidance or direction would be > appreciated. > > -Albert C. > > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trying to diagnose incomplete file transfer
Albert Croft via rsync wrote: > ... I am currently using the 'split' command to break the files > into 1-GB "chunks" (ex.: foo.tgz.aa, foo.tgz.ab, ...). > ... > I am frequently encountering times where the file appears to > have been transferred but is incomplete. (Example: foo.tgz.ab > now exists on the local system, has been removed from the remote, > but is incomplete.) One thing to check, not in rsync itself but in the preparation of the data: not all versions of "split" support files that aren't text. In particular, some will silently drop null bytes. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trying to diagnose incomplete file transfer
Hi. On Sat, 04 Mar 2023 00:39:52 -0600 Albert Croft via rsync wrote: > The rsync commands may be launched from command-line or cron, but use > the same format and options in either case. As a result, there may be > multiple rsync processes pulling files from the same remote path to the > same local path. I think you should first prevent this to happen. If your receiving machine is using systemd: - define a X.service for doing the rsync - define a X.timer unit to replace using cron - launch from the command line with: systemctl start X - this will not start a new rsync if one runs already (if X.service is running) -- francis -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
