On Thu, May 31, 2001 at 03:29:42PM -0700, Adam McKenna wrote:
> Background:
>
> Our web application picks up files from a repository and processes them --
> the repository is accessed by clients using SFTP. There is an rsync script
> running once per minute to sync up the client's data on the SFTP server with
> the "real" repository on our application servers.
>
> The Problem:
>
> Since the script runs once a minute, it is possible for the rsync to start
> while a file is still being transferred to us. What appears to happen at
> this time, is that something happens and the file doesn't finish transferring
> properly.
>
> So, is there any way to have rsync skip certain files, if they are not
> complete? There doesn't appear to be any obvious way to do this. Could
> the rsync process be the reason that these transfers are not finishing
> correctly?
>
> Thanks for any and all input.
>
> CC's would be appreciated as I am not a list subscriber -- but I will be
> checking the archive as well.
There is really no way around that problem with rsync. Many other people
have tried to do similar things and the wisdom on the mailing list has
always been that rsync via cron is the wrong tool for applications that
have data that changes frequently and needs to be replicated very soon
after it changes. You need something more like a distributed database or a
replicating filesystem, or at the very least call rsync in a synchronous
manner right after a file has been changed and before the file can change
again.
- Dave Dykstra
