[Bug 2647] --exclude options work from commandline but not in script
https://bugzilla.samba.org/show_bug.cgi?id=2647 [EMAIL PROTECTED] changed: What|Removed |Added Status|NEW |RESOLVED Resolution||WORKSFORME --- Additional Comments From [EMAIL PROTECTED] 2005-04-27 01:55 --- Rsync doesn't care about being run from inside a script or not. Depending on how the script was run, there can be differences in environment variables, but nothing in the command you mentioned should be affected by that (AFAICS). You should be looking for something in the script that is not quite what you think it is -- some subtle difference between the command you ran manually and the actual command in the script (such as a different source dir, a misspelled exclude, etc.). If you can't find what's wrong, feel free to attach the script to this bug report. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2654] timeout is always triggered with 2.6.4
https://bugzilla.samba.org/show_bug.cgi?id=2654 [EMAIL PROTECTED] changed: What|Removed |Added Status|NEW |ASSIGNED --- Additional Comments From [EMAIL PROTECTED] 2005-04-27 02:21 --- The generator's setting of the ignore_timeout variable is what allows the I/O code to continue updating last_io (the old method of clearing io_timeout in the generator used to leave last_io unchanged, which is no longer desireable). How short is your timeout? Are you using a delete option? I tried some tests, and couldn't duplicat the problem, so I'll need more details. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2654] timeout is always triggered with 2.6.4
https://bugzilla.samba.org/show_bug.cgi?id=2654 --- Additional Comments From [EMAIL PROTECTED] 2005-04-27 02:58 --- (In reply to comment #1) Oh, and one more thing: I am not longer sure if the bug did not exist in pre-2.6.4 versions. I did not see it before, but I had a mass-change with the side effect of the data being appended to my files. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2654] timeout is always triggered with 2.6.4
https://bugzilla.samba.org/show_bug.cgi?id=2654
--- Additional Comments From [EMAIL PROTECTED] 2005-04-27 02:54 ---
(In reply to comment #1)
The generator's setting of the ignore_timeout variable is what allows the I/O
code to continue updating last_io (the old method of clearing io_timeout in
the
generator used to leave last_io unchanged, which is no longer desireable).
Yup, I was wrong with my first guess. I tried modifying the source to reset
last_io when ignore_timeout is set to 1 and the error did not go away. last_io
DOES get modified, but there is more than one last_io, as I found out.
How short is your timeout? Are you using a delete option? I tried some
tests,
and couldn't duplicat the problem, so I'll need more details.
The timeout was big enough (3600), there was no delete, but a fuzzy option
specified. The files transferred were huge (800 MByte over a 512 KBit/s link)
and had been changed only in the last portion (data was appended). The timeout
did not matter, BTW, when I used 300 (and 1200) seconds, the timeout occured
shortly after that time, too.
To isolate the problem, I added rprintf() to the statements were last_io is
being modified. Since there are always two rsync client processes, which hold an
instance of last_io each, it looks that there were long sequences of
read_timeout() activity which were not interspersed by writefd_unbuffered() with
my kind of files.
I could see that last_io was set in read_timeout() over and over again and that
when check_timeout() was getting called, its instance of last_io (which
definitely had been set by writefd_unbuffered, as I could glean from the value)
was outdated.
I think this has to do with these statements, which I don't completely
understand:
read_timeout():
if (io_timeout fd == sock_f_in)
last_io = time(NULL);
writefd_unbuffered():
if (fd == sock_f_out) {
if (io_timeout)
last_io = time(NULL);
sleep_for_bwlimit(ret);
}
Each side (i.e. process) seems to watch only one direction of the data flow?
It seems since both sides are separated, too long pauses can arise in special
situations (such as mine) because data flows unidirectionally into the
unwatched direction for the specific process side and its last_io instance.
--
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
--- You are receiving this mail because: ---
You are the QA contact for the bug, or are watching the QA contact.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2654] timeout is always triggered with 2.6.4
https://bugzilla.samba.org/show_bug.cgi?id=2654 --- Additional Comments From [EMAIL PROTECTED] 2005-04-27 08:39 --- You mentioned using --fuzzy: are there a lot of files in your destination directories? One thing I've noticed with --fuzzy is that if rsync puts a lot of new files into a directory with thousands of old files, the name-scoring algorithm can chew up a lot of CPU. It might be possible for the connection to timeout if this slowed rsync down enough. (The --fuzzy performance into large dirs should be improved at some point.) As for who counts what, the generator only sends data down the socket, so it only notes that data in the last_io time. The receiver only reads data from the socket, so it only notes that data in the last_io time. The sender both reads and writes data from/to the socket, so it notes both. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2654] timeout is always triggered with 2.6.4
https://bugzilla.samba.org/show_bug.cgi?id=2654 --- Additional Comments From [EMAIL PROTECTED] 2005-04-27 09:26 --- (In reply to comment #4) About --fuzzy: I doubt that this was the case. There are only up to 50 files in each directory (and from what I understood, the fuzzy matching takes place only in the file's own directory). Also, there was ongoing socket activity a few seconds before the actual timeout occured. I saw that last_io was modified every second or so in read_timeout() while the received file got bigger. Then at some point, the check_timeout failed with an older last_io time. That timestamp matched that of the last modification in write_unbuffered(). As for who counts what, the generator only sends data down the socket, so it only notes that data in the last_io time. The receiver only reads data from the socket, so it only notes that data in the last_io time. The sender both reads and writes data from/to the socket, so it notes both. Yes. The sender was an rsync daemon in my case, whereas the client was separated into a generator and a receiver. Could that lead to a problem when only binary data (i.e. no deltas) are being sent (or the opposite, where just copying on the receiving side was neccessary)? Is there a possibility that the receiver is busy (modifying its last_io all the time) but the generator sits and waits for the next file to get processed or a select() timeout occurs and then the generator runs into the timeout in check_timeout() because the last time that write_unbuffered() modified its last_io was too long ago? This would mean that effectively, --timeout ought to large enough to accommodate not only pauses arising from directory scans, checksum calculations or fuzzy matching, but even data transfer itself if no (or maybe only) deltas are present. So, --timeout must in fact be longer than the expected transfer time of the largest file over a given connection in the worst case. I would expect --timeout to be a kind of inactivity counter to recognize stalled IP connections. But it seems that by separating the generator and receiver, there can be LONG pauses in one direction which trigger the timeout, although the connection is alive and well in the other direction (but that fact goes unnoticed). I guess in order to enable such bahaviour, the condition io_timeout fd == sock_f_in respectively io_timeout fd == sock_f_out should read something like io_timeout (fd == sock_f_in || fd == sock_f_out) in order to catch ACK packets? If the receiver and generator see only one direction of the same socket, however, only a kind of shared-memory variable last_io could do that trick. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--delete option does not always work
I know that there are a million messages from newbies who cannot get --delete to work. This message is special. :) --delete works fine for me EXCEPT when the following two conditions are present: 1) I am using the --relative option, and 2) the deleted file is in a subdirectory of the SRC directory. Here is a demonstration of how to recreate this bug: [EMAIL PROTECTED]:/tmp$ rsync --version rsync version 2.6.4 protocol version 29 Copyright (C) 1996-2005 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, IPv6, 64-bit system inums, 64-bit internal inums rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. [EMAIL PROTECTED]:/tmp$ mkdir -p src/1 [EMAIL PROTECTED]:/tmp$ cd src/1 [EMAIL PROTECTED]:/tmp/src/1$ touch a b c [EMAIL PROTECTED]:/tmp/src/1$ cd /tmp [EMAIL PROTECTED]:/tmp$ /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded -v src/ dest/ building file list ... done created directory dest src/ src/1/ src/1/a src/1/b src/1/c sent 235 bytes received 98 bytes 666.00 bytes/sec total size is 0 speedup is 0.00 [EMAIL PROTECTED]:/tmp$ rm src/1/c [EMAIL PROTECTED]:/tmp$ /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded -v src/ dest/ building file list ... done src/1/ sent 99 bytes received 26 bytes 250.00 bytes/sec total size is 0 speedup is 0.00 Now at this point we would expect there to be only files 'a' and 'b' to be present in /tmp/dest/src/1. However you can see that all three files are in that directory: [EMAIL PROTECTED]:/tmp$ ls -lR dest dest: total 1 drwxr-xr-x 3 lenny lenny 1024 Apr 27 12:37 src dest/src: total 1 drwxr-xr-x 2 lenny lenny 1024 Apr 27 12:38 1 dest/src/1: total 0 -rw-r--r-- 1 lenny lenny 0 Apr 27 12:37 a -rw-r--r-- 1 lenny lenny 0 Apr 27 12:37 b -rw-r--r-- 1 lenny lenny 0 Apr 27 12:37 c Just so you know, if you attempt to do this by placing files 'a','b', and 'c' just under the 'src' directory (without the '1' subdirectory), the --delete option will work correctly. Any thoughts? Thanks, Lenny -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2654] timeout is always triggered with 2.6.4
https://bugzilla.samba.org/show_bug.cgi?id=2654 --- Additional Comments From [EMAIL PROTECTED] 2005-04-27 10:47 --- What may have happened is that the outgoing socket from the generator may have filled up completely, and the timeout on outputting new data then got exceeded waiting for the other two processes to clear out enough space to write more. I think my decision to partially enable timeouts in the generator now that we're generating ACK packets was not the right choice. Try changing this code: if (protocol_version 29) ignore_timeout = 1; to remove the if and I think the timeout will go away. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete option does not always work
On Wed, Apr 27, 2005 at 12:57:51PM -0400, Lenny wrote: rsync -a --delete --numeric-ids --relative --delete-excluded -v src/ dest/ The problem is caused by the source dir being src/ instead of src when --relative is specified (this causes a problem internally because the starting directory is transferred as src/., so the receiver's heuristic to separate implied dirs in the file list from real dirs is not properly marking those subdirs as a part of the transferred hierarchy). You can work around the problem for now by dropping the trailing slash (while I work up a fix). Thanks for the report, ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --delete option does not always work
OK, I had a free moment to work up a fix, and here it is. It was
pretty simple.
..wayne..
--- flist.c 30 Mar 2005 17:31:35 - 1.291
+++ flist.c 27 Apr 2005 19:14:17 -
@@ -655,6 +655,9 @@ static struct file_struct *receive_file_
if (flags XMIT_TOP_DIR) {
in_del_hier = 1;
del_hier_name_len = file-dir.depth == 0 ? 0 : l1 + l2;
+ if (relative_paths del_hier_name_len 2
+basename_len == 1+1 *basename == '.')
+ del_hier_name_len -= 2;
file-flags |= FLAG_TOP_DIR | FLAG_DEL_HERE;
} else if (in_del_hier) {
if (!relative_paths || !del_hier_name_len
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2654] timeout is always triggered with 2.6.4
https://bugzilla.samba.org/show_bug.cgi?id=2654 --- Additional Comments From [EMAIL PROTECTED] 2005-04-27 14:08 --- (In reply to comment #6) generating ACK packets was not the right choice. Try changing this code: if (protocol_version 29) ignore_timeout = 1; to remove the if and I think the timeout will go away. That did it for me. No timeouts any more. So the bug was present only since 2.6.4 in fact. I think this one may have gone unnoticed by now since few people synchronize that much data over semi-permanent connections (i.e. dynamic IPs), so that line drops have to be detected by the --timeout option. Also, the up- and downstreams differ greatly (ADSL) in my setup, so your assumptions about queues seem logical. Thanks for the prompt fixing! -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2654] timeout is always triggered with 2.6.4
https://bugzilla.samba.org/show_bug.cgi?id=2654 [EMAIL PROTECTED] changed: What|Removed |Added Status|ASSIGNED|RESOLVED Resolution||FIXED --- Additional Comments From [EMAIL PROTECTED] 2005-04-27 15:59 --- Excellent! I've checked in a fix for this (and a big comment about why we like turning off the timeouts in the generator). -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Wed Apr 27 22:56:47 2005 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv18703 Modified Files: flist.c Log Message: If the user specified --relative and a source dir with a trailing slash, make sure we strip of the trailing /. from the dir's len (in del_hier_name_len) so that its subdirs get marked with FLAG_DEL_HERE. Revisions: flist.c 1.291 = 1.292 http://www.samba.org/cgi-bin/cvsweb/rsync/flist.c?r1=1.291r2=1.292 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Wed Apr 27 22:58:07 2005 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv19975 Modified Files: generator.c Log Message: Set ignore_timeout before starting the transfer, regardless of what protocol level is in effect. This guards against the generator timing out when the output pipe is full and the input pipe (to the receiver) is slow. Revisions: generator.c 1.207 = 1.208 http://www.samba.org/cgi-bin/cvsweb/rsync/generator.c?r1=1.207r2=1.208 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
