2.6.9 and timeouts
I have a scenario where a broken pipe condition can happen on a fairly regular basis and in all honesty it is to be expected. We are working with satellite links to moving ship platforms (no it isn't military based) and depending on location, sea conditions, and a few dozen other factors, the data link can become somewhat unstable at times. That and the high-latency for IP traffic itself (600ms to 1500ms) over such a link has always presented some challenges for transfering/syncing data. So we accept the fact that even with acceleration devices and other methods to lower latency, the bottom line is you can still have some very small to large interruptions in service (and again, depends on the environment at that time). To try and avoid some of the rsync broken pipe errors we were trying to use a timeout setting of 600 (10 minutes) with 2.6.9. This does help but still presents some failures. I was starting to write a wrapper script that checked for these timeouts and perform a wait+retry and noticed something odd. The seconds being reported in the error message was signifigantly higher than 600. 3 Consecutive Examples: $ rsync -av --progress --timeout=600 --partial tfile.tgz host1:/xfr building file list ... 1 file to consider tfile.tgz io timeout after 1528 seconds -- exiting rsync error: timeout in data send/receive (code 30) at io.c(165) [sender=2.6.9] $ rsync -av --progress --timeout=600 --partial tfile.tgz host1:/xfr building file list ... 1 file to consider tfile.tgz io timeout after 9930 seconds -- exiting rsync error: timeout in data send/receive (code 30) at io.c(165) [sender=2.6.9] $ rsync -av --progress --timeout=600 --partial tfile.tgz host1:/xfr building file list ... 1 file to consider tfile.tgz io timeout after 4825 seconds -- exiting rsync error: timeout in data send/receive (code 30) at io.c(165) [sender=2.6.9] The 1528, 9930, and 4825 seconds is what caught my eye. They could actually be correct as to when progress stopped vs. when it timed out, but what happened to timeout=600? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Using --remove-source-files with backup?
Many, Many Thanks for this Matt! Seems to be working great so far. Just one question though. My initial tests with --backup-dir and -b had just the file being backed up to the specified dir. With --source-backup the tree structure of the original source file is also being included in the --backup-dir directory. example: rsync --remove-source-files --source-backup --backup-dir=/arc -av /home/dir/file remotehost:/dir/file results in the backup file being put in /arc/home/dir/file instead of /arc/file. It isn't a problem for me as my tree structure is limited to just a couple of directories, but could be a problem for those with a deeper one. Or am I not specifying something correctly? Again, many thanks for this. -Scott Original Message Subject: Re: Using --remove-source-files with backup? From: Matt McCutchen [EMAIL PROTECTED] Date: Sun, May 06, 2007 6:38 am To: ScottZ [EMAIL PROTECTED] Cc: rsync@lists.samba.org On 5/6/07, Matt McCutchen [EMAIL PROTECTED] wrote: The attached patch to rsync 2.6.9 adds a --source-backup option that backs up removed source files. Oops, I made a typo in the addition to the man page. The fixed patch is attached. Matt -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Using --remove-source-files with backup?
Essentially you want to move each source file to another directory after it is successfully transferred. Exactly. I'm sure there's a way to accomplish this, but it might not be simple. I'll tell you if I find one. Thanks for looking at this Matt. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Using --remove-source-files with backup?
I have a scenario where I am copying from a local system to remote target and need to remove the local source files once they have been successfully transfered. --remove-source-files works fine for this. An issue has come up in that I need to be able to backup the removed local source files as well. With rsync 2.6.9 I tried: rsync --remove-source-files --backup-dir=/archive abv (source) (remotehost:target) The result was that the backup files were made in /archive on the remotehost system and not the local system (where the original files were removed). Is this a bug or is the backup option only able to backup deleted target files? Is there another way around this? Note: As a WAG, I tried using a myhost:/archive (where myhost is my source host) type of entry for --backup-dir, but that just created a myhost: directory on the remotehost:target directory. Like I said, it was a WAG. Thanks, -Scott -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
