[Fwd: Re: rsync windows - unix still hanging :(]
Author of the message didn't include [EMAIL PROTECTED] in the reply,
and I think this message is in topic.
Original Message
Subject: Re: rsync windows - unix still hanging :(
Date: Mon, 30 Dec 2002 16:10:32 -0800
From: Jim Kleckner [EMAIL PROTECTED]
To: Mike Rubel [EMAIL PROTECTED], [EMAIL PROTECTED]
Mike -
Greger Cronquist and I have been using a patched rsync
successfully. This was patched from two published patches,
one for buffering which seems to cause severe performance
problems under cygwin while only mild under Linux. The
is for the hang documented elsewhere. Here are patch discussion links:
http://www.cygwin.com/ml/cygwin/2002-10/msg00308.html
http://sources.redhat.com/ml/cygwin/2002-09/msg01155.html
Below are the diffs. Note the use of 100ms not 30ms as per the
original posting.
Since you are on the rsync list, can you get them into that code line?
Jim
On Sat, 28 Dec 2002, Scott Evans wrote:
I spent about 12 hours today writing a nice backup system based on
Mike Rubel's snapshot system; but much to my dismay, when I took it
for a test drive, I found that rsync hung on the first large directory
I tried to back up. Aiee!
[ ... ]
Hey Scott,
I'm Mike Rubel (author of that snapshot system page)--I haven't heard
about this cygwin issue before, but this would definitely be worth adding
to the FAQ. I'm sorry to hear you're having so much trouble with it!
You're actually the second person to mention windows issues; the other
fellow was using a SAMBA mount and rsyncing locally on the server. The
problem there was files mysteriously getting copied when they hadn't
changed, or deleted when they weren't supposed to be. I asked the
list if
anyone else had seen that behavior, but no one responded, so I'm not sure
what to think. I'll add it to the FAQ, anyway, and maybe the BUGS
section.
Please let me know if you find any additional information. I'm on the
rsync list but not the cygwin list (yet).
I've had no trouble with this at home with my roommate's Win2K PC, but
between this case and the previous one, it sounds like the rsync/windows
interaction may still have some rough edges, which is not really
acceptable for a backup solution. :(
Thanks,
Here are the patches/diffs against a freshly cvs updated rsync cvs tree:
Index: fileio.c
===
RCS file: /cvsroot/rsync/fileio.c,v
retrieving revision 1.5
diff -r1.5 fileio.c
79c79,107
return write(f,buf,len);
---
static char *writeBuf;
static size_t writeBufSize;
static size_t writeBufCnt;
if ( !writeBuf ) {
writeBufSize = MAX_MAP_SIZE;
writeBufCnt = 0;
writeBuf = (char*)malloc(MAX_MAP_SIZE);
if (!writeBuf) out_of_memory(write_file);
}
ret = len;
do {
if ( buf writeBufCnt writeBufSize ) {
size_t copyLen = len;
if ( copyLen writeBufSize - writeBufCnt ) {
copyLen = writeBufSize - writeBufCnt;
}
memcpy(writeBuf + writeBufCnt, buf, copyLen);
writeBufCnt += copyLen;
buf += copyLen;
len -= copyLen;
}
if ( !buf || writeBufCnt == writeBufSize ) {
int thisRet = write(f, writeBuf, writeBufCnt);
if ( thisRet 0 ) return thisRet;
writeBufCnt = 0;
}
} while ( buf len 0 );
return ret;
Index: flist.c
===
RCS file: /cvsroot/rsync/flist.c,v
retrieving revision 1.127
diff -r1.127 flist.c
894c894
io_start_buffering(f);
---
io_start_buffering_out(f);
Index: io.c
===
RCS file: /cvsroot/rsync/io.c,v
retrieving revision 1.105
diff -r1.105 io.c
44,45c44,45
static int multiplex_in_fd;
static int multiplex_out_fd;
---
static int multiplex_in_fd = -1;
static int multiplex_out_fd = -1;
288a289,291
static char *buffer;
static size_t bufferIdx = 0;
static size_t bufferSz;
290c293
if (!io_multiplexing_in || fd != multiplex_in_fd)
---
if (fd != multiplex_in_fd)
292a296,305
if (!io_multiplexing_in remaining == 0) {
if (!buffer) {
bufferSz = 2 * IO_BUFFER_SIZE;
buffer = malloc(bufferSz);
if (!buffer) out_of_memory(read_unbuffered);
}
remaining = read_timeout(fd, buffer, bufferSz);
bufferIdx = 0;
}
296c309,310
read_loop(fd, buf, len);
---
memcpy(buf, buffer + bufferIdx, len);
bufferIdx += len;
299c313
continue;
---
break;
308c322,329
if (tag == MPLEX_BASE)
---
if (tag == MPLEX_BASE) {
if (!buffer || remaining bufferSz) {
buffer = Realloc(buffer, remaining);
if
[Fwd: Re: rsync windows - unix still hanging :(]
Author of the message didn't include [EMAIL PROTECTED] in the reply,
and I think this message is in topic.
Original Message
Subject: Re: rsync windows - unix still hanging :(
Date: Mon, 30 Dec 2002 17:24:47 -0800
From: Jim Kleckner [EMAIL PROTECTED]
To: Mike Rubel [EMAIL PROTECTED]
CC: [EMAIL PROTECTED]
References:
[EMAIL PROTECTED]
The msleep(100) call is not checked into version 1.156 of main.c.
I would recommend trying this one-line patch to see if it helps.
I haven't personally delved into the true meaning.
I just checked my read-only CVS tree for rsync again.
Here is the context diff for that file. Note that the msleep(100)
is not in the checked in version.
Jim
Index: main.c
===
RCS file: /cvsroot/rsync/main.c,v
retrieving revision 1.156
diff -c -r1.156 main.c
*** main.c1 Aug 2002 20:46:59 -1.156
--- main.c31 Dec 2002 01:21:34 -
***
*** 346,351
--- 346,353
exit_cleanup(0);
}
+ io_start_buffering_in(f_in);
+ io_start_buffering_out(f_out);
send_files(flist,f_out,f_in);
io_flush();
report(f_out);
***
*** 421,427
close(error_pipe[1]);
if (f_in != f_out) close(f_in);
! io_start_buffering(f_out);
io_set_error_fd(error_pipe[0]);
--- 423,429
close(error_pipe[1]);
if (f_in != f_out) close(f_in);
! io_start_buffering_out(f_out);
io_set_error_fd(error_pipe[0]);
***
*** 434,440
write_int(f_out, -1);
}
io_flush();
!
kill(pid, SIGUSR2);
wait_process(pid, status);
return status;
--- 436,442
write_int(f_out, -1);
}
io_flush();
! msleep(100);
kill(pid, SIGUSR2);
wait_process(pid, status);
return status;
***
*** 476,481
--- 478,484
}
}
+ io_start_buffering_in(f_in);
if (delete_mode !delete_excluded)
recv_exclude_list(f_in);
***
*** 569,574
--- 572,578
extern int cvs_exclude;
extern int delete_mode;
extern int delete_excluded;
+ io_start_buffering_out(f_out);
if (cvs_exclude)
add_cvs_excludes();
if (delete_mode !delete_excluded)
***
*** 578,584
--- 582,591
if (verbose 3)
rprintf(FINFO,file list sent\n);
+ io_flush();
+ io_start_buffering_out(f_out);
send_files(flist,f_out,f_in);
+ io_flush();
if (remote_version = 24) {
/* final goodbye message */
read_int(f_in);
***
*** 590,595
--- 597,603
wait_process(pid, status);
}
report(-1);
+ io_flush();
exit_cleanup(status);
}
Mike Rubel wrote:
[ ... ]
Could you clarify for me whether the patch Steve refers to is the same as
(or has the same effect as) the one you describe below? Or are they two
different animals? If they are different, should we test whether this
patch fixes his specific problem?
Best regards,
Mike
http://www.mikerubel.org
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Bug reporting: http://cygwin.com/bugs.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
--
Lapo 'Raist' Luchini
[EMAIL PROTECTED] (PGP X.509 keys available)
http://www.lapo.it (ICQ UIN: 529796)
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
»¶Ó¶©ÔÄ¡¶ÎÒºÜÇîÒ×¾Ô¤²â×ã²Ê×ÊÁÏ¡·
21£¬25ÆÚÈ«ÖУ¬22£¬23£¬30£¬31ÖÐ12³¡¡£ÆäÓàÖÐ10-11³¡¡£ [EMAIL PROTECTED] QQ:18419276 »¶Ó¹âÁÙÍøÉÏÓªÏúÈí¼þ³¬ÊÐ http://www.sesoft.athissite.com ±¾ÓʼþÓ󬼶¹ã¸æÈí¼þ·¢ËÍ http://www.sesoft.athissite.com -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
rsync feature suggestion
Hi, I'd like to suggest a new feature to rsync. Problem: Currently, rsync generates a recursive list of file existing a the source directory, modifies this list by includes and excludes, and then copies these files. That's pretty good in most, but not all cases. I am mirroring a debian archive, but unfortunately, debian mixes all files of several distributions in a subtree /pool. There is no way to select only the files of a certain distribution through a simple exclude/include expression. There is a tool called debmirror, which first downloads the distribution index files, extracts all the filenames/paths of the files needed and then calls rsync for every single file. Thats certainly not useful, especially since rsync shows the servers motd for every single file. Therefore, I'd like to suggest a new option: Allow rsync to not build the list of files existing at the source directory by recursively walking through the source directory, but by reading a file or stdin to get a list of files to be copied. This would allow to mirror the distribution index files in a first step, then build the list of files needed and then to download these files is a second step. An alternative method would be to keep the recursive method, but to open a pipe to an external program. Before downloading a file, the path is printed to the pipe and an answer is read from the pipe. Thus, an external filter program can decide for each single file whether to copy it or not. regards Hadmut (Please respond directly, I'm not on your mailing list) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [Fwd: Re: rsync windows - unix still hanging :(]
Author of the message didn't include [EMAIL PROTECTED] in the reply,
and I think this message is in topic.
Yes, thanks for cc'ing the rsync list. The net result of all this,
btw, was that I tried the patches Jim suggested and they didn't solve
the problem for me (I'm the guy who started the thread) -- rsync still
consistently hangs on large-ish directories.
scott
Original Message
Subject: Re: rsync windows - unix still hanging :(
Date: Mon, 30 Dec 2002 17:24:47 -0800
From: Jim Kleckner [EMAIL PROTECTED]
To: Mike Rubel [EMAIL PROTECTED]
CC: [EMAIL PROTECTED]
References:
[EMAIL PROTECTED]
The msleep(100) call is not checked into version 1.156 of main.c.
I would recommend trying this one-line patch to see if it helps.
I haven't personally delved into the true meaning.
I just checked my read-only CVS tree for rsync again.
Here is the context diff for that file. Note that the msleep(100)
is not in the checked in version.
Jim
Index: main.c
===
RCS file: /cvsroot/rsync/main.c,v
retrieving revision 1.156
diff -c -r1.156 main.c
*** main.c1 Aug 2002 20:46:59 -1.156
--- main.c31 Dec 2002 01:21:34 -
***
*** 346,351
--- 346,353
exit_cleanup(0);
}
+ io_start_buffering_in(f_in);
+ io_start_buffering_out(f_out);
send_files(flist,f_out,f_in);
io_flush();
report(f_out);
***
*** 421,427
close(error_pipe[1]);
if (f_in != f_out) close(f_in);
! io_start_buffering(f_out);
io_set_error_fd(error_pipe[0]);
--- 423,429
close(error_pipe[1]);
if (f_in != f_out) close(f_in);
! io_start_buffering_out(f_out);
io_set_error_fd(error_pipe[0]);
***
*** 434,440
write_int(f_out, -1);
}
io_flush();
!
kill(pid, SIGUSR2);
wait_process(pid, status);
return status;
--- 436,442
write_int(f_out, -1);
}
io_flush();
! msleep(100);
kill(pid, SIGUSR2);
wait_process(pid, status);
return status;
***
*** 476,481
--- 478,484
}
}
+ io_start_buffering_in(f_in);
if (delete_mode !delete_excluded)
recv_exclude_list(f_in);
***
*** 569,574
--- 572,578
extern int cvs_exclude;
extern int delete_mode;
extern int delete_excluded;
+ io_start_buffering_out(f_out);
if (cvs_exclude)
add_cvs_excludes();
if (delete_mode !delete_excluded)
***
*** 578,584
--- 582,591
if (verbose 3)
rprintf(FINFO,file list sent\n);
+ io_flush();
+ io_start_buffering_out(f_out);
send_files(flist,f_out,f_in);
+ io_flush();
if (remote_version = 24) {
/* final goodbye message */
read_int(f_in);
***
*** 590,595
--- 597,603
wait_process(pid, status);
}
report(-1);
+ io_flush();
exit_cleanup(status);
}
Mike Rubel wrote:
[ ... ]
Could you clarify for me whether the patch Steve refers to is the same as
(or has the same effect as) the one you describe below? Or are they two
different animals? If they are different, should we test whether this
patch fixes his specific problem?
Best regards,
Mike
http://www.mikerubel.org
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Bug reporting: http://cygwin.com/bugs.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync feature suggestion
Might it be possible to take the file list that you want to feed to
rsync and turn it into an rsync.conf file?
A simple bash script could create the config file and call rsync (with
the --config= to specify the temporary config file)
Something like this (syntax most likely is wrong, haven't tested it):
#!/bin/sh
IFS=
cat /etc/rsync.conf rsync_command
FILES_TO_SYNC=`cat file_list.txt`
for EACH_FILE in $FILES_TO_SYNC; do
echo ' --include=${EACH_FILE}' rsync_command
done
rsync --config=rsync_command
- Ed King
Hadmut Danisch wrote:
Hi,
I'd like to suggest a new feature to rsync.
Problem:
Currently, rsync generates a recursive list of file
existing a the source directory, modifies this list by
includes and excludes, and then copies these files.
That's pretty good in most, but not all cases.
I am mirroring a debian archive, but unfortunately,
debian mixes all files of several distributions in a
subtree /pool. There is no way to select only the files
of a certain distribution through a simple exclude/include
expression.
There is a tool called debmirror, which first downloads
the distribution index files, extracts all the filenames/paths
of the files needed and then calls rsync for every single file.
Thats certainly not useful, especially since rsync shows the
servers motd for every single file.
Therefore, I'd like to suggest a new option: Allow rsync to
not build the list of files existing at the source directory
by recursively walking through the source directory, but by
reading a file or stdin to get a list of files to be copied.
This would allow to mirror the distribution index files in a
first step, then build the list of files needed and then to
download these files is a second step.
An alternative method would be to keep the recursive method, but
to open a pipe to an external program. Before downloading a
file, the path is printed to the pipe and an answer is read
from the pipe. Thus, an external filter program can decide for
each single file whether to copy it or not.
regards
Hadmut
(Please respond directly, I'm not on your mailing list)
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync feature suggestion
Hadmut Danisch wrote: I'd like to suggest a new feature to rsync. I am mirroring a debian archive, but unfortunately, debian mixes all files of several distributions in a subtree /pool. There is no way to select only the files of a certain distribution through a simple exclude/include expression. There is a tool called debmirror, which first downloads the distribution index files, extracts all the filenames/paths of the files needed and then calls rsync for every single file. Thats certainly not useful, especially since rsync shows the servers motd for every single file. I was about to suggest: $ rsync --include-from=list-file --exclude=\* but of course that will exclude the parent directories of files you want, causing them to be ignored. This might work: $ rsync --include-from=list-file --include=\*\*/ --exclude=\* although it will mirror the entire directory structure (but not unspecified files). Probably, rsync should be taught that: If I explicitly include a file, look for it explicitly, even if I've excluded a parent directory. Max. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync feature suggestion
Max Bowsher wrote Hadmut Danisch wrote: I'd like to suggest a new feature to rsync. I am mirroring a debian archive, but unfortunately, debian mixes all files of several distributions in a subtree /pool. There is no way to select only the files of a certain distribution through a simple exclude/include expression. There is a tool called debmirror, which first downloads the distribution index files, extracts all the filenames/paths of the files needed and then calls rsync for every single file. Thats certainly not useful, especially since rsync shows the servers motd for every single file. I was about to suggest: $ rsync --include-from=list-file --exclude=\* but of course that will exclude the parent directories of files you want, causing them to be ignored. This might work: $ rsync --include-from=list-file --include=\*\*/ --exclude=\* although it will mirror the entire directory structure (but not unspecified files). Probably, rsync should be taught that: If I explicitly include a file, look for it explicitly, even if I've excluded a parent directory. Not too long ago, I modified/mangled rsync to do rsync --files-from /some/file --include-regexes /some/regular/expressions \ --exclude-regexes /some/regular/expressions such that all the files in /some/file would be sent iff they matched the posix regexes in --include-regexes and didn't match the ones in --exclude-regexes (if present). I don't have a wide variety of platforms to test it on, but it worked okay on linux, solaris, and irix. -justinb -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync feature suggestion
On Fri, Jan 03, 2003 at 11:28:37AM -0600, Edward King wrote: Might it be possible to take the file list that you want to feed to rsync and turn it into an rsync.conf file? It might be possible, but maybe it is ambiguous and definitely not efficient, since --include defines a Pattern, not a file name/path. As far as I know, rsync has to check every single file against all include/exclude patterns. That's a complexity of O(n^2). I'm talking about directories with 30,000 .. 1,000,000 files. This could end up in 10^12 file name/comparison patterns, and that's certainly not what you want to have. If you read a list of plain filenames, you do not need to perfom pattern matching, but can use a simple associative/hash array and check extremely fast, whether a given filename is to be copied or not. That's a very important difference to a list of patterns. regards Hadmut -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync feature suggestion
Hi, I just sent an answer to Edward's similar suggestion to the list. regards Hadmut -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync windows - unix still hanging :(
I am seeing similar symptoms. -- Compaq 350 mhz CPU Top shows memory at: Mem: 154840K av, 25904K used, 128936K free, 0K shrd,2532K buff Swap: 264592K av, 0K used, 264592K free 12004K cached Network is 10mb hub connection. Running: rsync version 2.5.4 protocol version 26 On a redhat 7.3 system, patched up to the current level. Updated Kernel 2.4.18-18.7.x #1 Wed Nov 13 19:30:43 EST -- Backing up Windows shares via samba mounts to a local dedicated ide disk (60mb). This solution works wonderfully, BTW. With one exception... Backing up a large volume of images that range in size from 18 mb to +100mb each. Approximately 45 GB total. Files are distributed in sub-directories, some large, but not overly excessive. Source machine is Windows 2000 server. Disk partition with approx 45 mb. The rsync proceeds nicely using: rsync -rtv /mnt/ntserver/sharename /mountedbackupdisk/ It produces expected results. However, when the command is run again, it finds two or three dozen files it thinks need to be refreshed. I stop the rsync, and check the file sizes, perms, and chksums. All are a match. If I allow it to refresh the files, and run the command again. It finds fewer to refresh. Subsequent runs will eventually cause the refreshes to stop. Odd behaviour. If I run: rsync -rtvvv /mnt/ntserver/sharename /mountedbackupdisk/ The /mnt/ntserver/sharename is a samba mounted win2000 filesystem. The output stops after about 1100 lines with: make_file(4,1mb/BMTT013.tif) The process is still there, but the load drops and there are no packets going in/out of the ethernet interface. I think rsync has hung. Note that this occurs only when I specify additional -vvv. I saw somewhat similar results when backing up the 4 office workstations, which run win98 and winxp. Re-running the command would find files that needed to be backed up. I thought these might be files that changed while backup was running, but running the command at night, I would see the same. Again, eventually, it would report no changes to be updated. I have used rsync for years to backup Linux boxes to disk. It works wonderfully! I want the same reliability and efficiency when backing up windows too. If anyone can help, I would really appreciate it. If I can provide more details or do some testing, I have quite a few boxes at my disposal. Thanks! -- Timothy Burt General Manager Arbor Group LLC Los Angeles, Calif. USA -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync windows - unix still hanging :(
On Fri, 3 Jan 2003, Timothy Burt wrote: I am seeing similar symptoms. Hi Timothy, I'm trying to find out more about this. May I ask, are the files you see sent even when they haven't changed regular files, or directories, or both? Mike -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
directories that change into symlinks
our daily backup is done using the rdiff-backup tool, which in turn utilizes rsync/librsync to do the actual mirroring work. a few days ago we did a refactoring and renamed a bunch of directories. for backward compatibility we maintain the old names by symlinking it to the new names. so, for example, oldname1/ now becomes newname1/, and oldname1 is now a symlink to newname1/. we found that now the mirroring cannot complete. rsync doesn't seem to be able to handle this. it tries to do an rmdir(oldname1) followed by symlink(newname1,oldname1). however, since the directory oldname1/ in the old mirror is not empty, rmdir fails with Directory not empty and thus symlink fails too with File exists (since oldname1 has not been deleted yet). any pointers? we looked at the available rsync options but have found no clue yet. -- dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
