[Fwd: Re: rsync windows - unix still hanging :(]
Author of the message didn't include [EMAIL PROTECTED] in the reply,
and I think this message is in topic.
Original Message
Subject: Re: rsync windows - unix still hanging :(
Date: Mon, 30 Dec 2002 16:10:32 -0800
From: Jim Kleckner [EMAIL PROTECTED]
To: Mike Rubel [EMAIL PROTECTED], [EMAIL PROTECTED]
Mike -
Greger Cronquist and I have been using a patched rsync
successfully. This was patched from two published patches,
one for buffering which seems to cause severe performance
problems under cygwin while only mild under Linux. The
is for the hang documented elsewhere. Here are patch discussion links:
http://www.cygwin.com/ml/cygwin/2002-10/msg00308.html
http://sources.redhat.com/ml/cygwin/2002-09/msg01155.html
Below are the diffs. Note the use of 100ms not 30ms as per the
original posting.
Since you are on the rsync list, can you get them into that code line?
Jim
On Sat, 28 Dec 2002, Scott Evans wrote:
I spent about 12 hours today writing a nice backup system based on
Mike Rubel's snapshot system; but much to my dismay, when I took it
for a test drive, I found that rsync hung on the first large directory
I tried to back up. Aiee!
[ ... ]
Hey Scott,
I'm Mike Rubel (author of that snapshot system page)--I haven't heard
about this cygwin issue before, but this would definitely be worth adding
to the FAQ. I'm sorry to hear you're having so much trouble with it!
You're actually the second person to mention windows issues; the other
fellow was using a SAMBA mount and rsyncing locally on the server. The
problem there was files mysteriously getting copied when they hadn't
changed, or deleted when they weren't supposed to be. I asked the
list if
anyone else had seen that behavior, but no one responded, so I'm not sure
what to think. I'll add it to the FAQ, anyway, and maybe the BUGS
section.
Please let me know if you find any additional information. I'm on the
rsync list but not the cygwin list (yet).
I've had no trouble with this at home with my roommate's Win2K PC, but
between this case and the previous one, it sounds like the rsync/windows
interaction may still have some rough edges, which is not really
acceptable for a backup solution. :(
Thanks,
Here are the patches/diffs against a freshly cvs updated rsync cvs tree:
Index: fileio.c
===
RCS file: /cvsroot/rsync/fileio.c,v
retrieving revision 1.5
diff -r1.5 fileio.c
79c79,107
return write(f,buf,len);
---
static char *writeBuf;
static size_t writeBufSize;
static size_t writeBufCnt;
if ( !writeBuf ) {
writeBufSize = MAX_MAP_SIZE;
writeBufCnt = 0;
writeBuf = (char*)malloc(MAX_MAP_SIZE);
if (!writeBuf) out_of_memory(write_file);
}
ret = len;
do {
if ( buf writeBufCnt writeBufSize ) {
size_t copyLen = len;
if ( copyLen writeBufSize - writeBufCnt ) {
copyLen = writeBufSize - writeBufCnt;
}
memcpy(writeBuf + writeBufCnt, buf, copyLen);
writeBufCnt += copyLen;
buf += copyLen;
len -= copyLen;
}
if ( !buf || writeBufCnt == writeBufSize ) {
int thisRet = write(f, writeBuf, writeBufCnt);
if ( thisRet 0 ) return thisRet;
writeBufCnt = 0;
}
} while ( buf len 0 );
return ret;
Index: flist.c
===
RCS file: /cvsroot/rsync/flist.c,v
retrieving revision 1.127
diff -r1.127 flist.c
894c894
io_start_buffering(f);
---
io_start_buffering_out(f);
Index: io.c
===
RCS file: /cvsroot/rsync/io.c,v
retrieving revision 1.105
diff -r1.105 io.c
44,45c44,45
static int multiplex_in_fd;
static int multiplex_out_fd;
---
static int multiplex_in_fd = -1;
static int multiplex_out_fd = -1;
288a289,291
static char *buffer;
static size_t bufferIdx = 0;
static size_t bufferSz;
290c293
if (!io_multiplexing_in || fd != multiplex_in_fd)
---
if (fd != multiplex_in_fd)
292a296,305
if (!io_multiplexing_in remaining == 0) {
if (!buffer) {
bufferSz = 2 * IO_BUFFER_SIZE;
buffer = malloc(bufferSz);
if (!buffer) out_of_memory(read_unbuffered);
}
remaining = read_timeout(fd, buffer, bufferSz);
bufferIdx = 0;
}
296c309,310
read_loop(fd, buf, len);
---
memcpy(buf, buffer + bufferIdx, len);
bufferIdx += len;
299c313
continue;
---
break;
308c322,329
if (tag == MPLEX_BASE)
---
if (tag == MPLEX_BASE) {
if (!buffer || remaining bufferSz) {
buffer = Realloc(buffer, remaining);
if
[Fwd: Re: rsync windows - unix still hanging :(]
Author of the message didn't include [EMAIL PROTECTED] in the reply,
and I think this message is in topic.
Original Message
Subject: Re: rsync windows - unix still hanging :(
Date: Mon, 30 Dec 2002 17:24:47 -0800
From: Jim Kleckner [EMAIL PROTECTED]
To: Mike Rubel [EMAIL PROTECTED]
CC: [EMAIL PROTECTED]
References:
[EMAIL PROTECTED]
The msleep(100) call is not checked into version 1.156 of main.c.
I would recommend trying this one-line patch to see if it helps.
I haven't personally delved into the true meaning.
I just checked my read-only CVS tree for rsync again.
Here is the context diff for that file. Note that the msleep(100)
is not in the checked in version.
Jim
Index: main.c
===
RCS file: /cvsroot/rsync/main.c,v
retrieving revision 1.156
diff -c -r1.156 main.c
*** main.c1 Aug 2002 20:46:59 -1.156
--- main.c31 Dec 2002 01:21:34 -
***
*** 346,351
--- 346,353
exit_cleanup(0);
}
+ io_start_buffering_in(f_in);
+ io_start_buffering_out(f_out);
send_files(flist,f_out,f_in);
io_flush();
report(f_out);
***
*** 421,427
close(error_pipe[1]);
if (f_in != f_out) close(f_in);
! io_start_buffering(f_out);
io_set_error_fd(error_pipe[0]);
--- 423,429
close(error_pipe[1]);
if (f_in != f_out) close(f_in);
! io_start_buffering_out(f_out);
io_set_error_fd(error_pipe[0]);
***
*** 434,440
write_int(f_out, -1);
}
io_flush();
!
kill(pid, SIGUSR2);
wait_process(pid, status);
return status;
--- 436,442
write_int(f_out, -1);
}
io_flush();
! msleep(100);
kill(pid, SIGUSR2);
wait_process(pid, status);
return status;
***
*** 476,481
--- 478,484
}
}
+ io_start_buffering_in(f_in);
if (delete_mode !delete_excluded)
recv_exclude_list(f_in);
***
*** 569,574
--- 572,578
extern int cvs_exclude;
extern int delete_mode;
extern int delete_excluded;
+ io_start_buffering_out(f_out);
if (cvs_exclude)
add_cvs_excludes();
if (delete_mode !delete_excluded)
***
*** 578,584
--- 582,591
if (verbose 3)
rprintf(FINFO,file list sent\n);
+ io_flush();
+ io_start_buffering_out(f_out);
send_files(flist,f_out,f_in);
+ io_flush();
if (remote_version = 24) {
/* final goodbye message */
read_int(f_in);
***
*** 590,595
--- 597,603
wait_process(pid, status);
}
report(-1);
+ io_flush();
exit_cleanup(status);
}
Mike Rubel wrote:
[ ... ]
Could you clarify for me whether the patch Steve refers to is the same as
(or has the same effect as) the one you describe below? Or are they two
different animals? If they are different, should we test whether this
patch fixes his specific problem?
Best regards,
Mike
http://www.mikerubel.org
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Bug reporting: http://cygwin.com/bugs.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
--
Lapo 'Raist' Luchini
[EMAIL PROTECTED] (PGP X.509 keys available)
http://www.lapo.it (ICQ UIN: 529796)
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: [Fwd: Re: rsync windows - unix still hanging :(]
Author of the message didn't include [EMAIL PROTECTED] in the reply,
and I think this message is in topic.
Yes, thanks for cc'ing the rsync list. The net result of all this,
btw, was that I tried the patches Jim suggested and they didn't solve
the problem for me (I'm the guy who started the thread) -- rsync still
consistently hangs on large-ish directories.
scott
Original Message
Subject: Re: rsync windows - unix still hanging :(
Date: Mon, 30 Dec 2002 17:24:47 -0800
From: Jim Kleckner [EMAIL PROTECTED]
To: Mike Rubel [EMAIL PROTECTED]
CC: [EMAIL PROTECTED]
References:
[EMAIL PROTECTED]
The msleep(100) call is not checked into version 1.156 of main.c.
I would recommend trying this one-line patch to see if it helps.
I haven't personally delved into the true meaning.
I just checked my read-only CVS tree for rsync again.
Here is the context diff for that file. Note that the msleep(100)
is not in the checked in version.
Jim
Index: main.c
===
RCS file: /cvsroot/rsync/main.c,v
retrieving revision 1.156
diff -c -r1.156 main.c
*** main.c1 Aug 2002 20:46:59 -1.156
--- main.c31 Dec 2002 01:21:34 -
***
*** 346,351
--- 346,353
exit_cleanup(0);
}
+ io_start_buffering_in(f_in);
+ io_start_buffering_out(f_out);
send_files(flist,f_out,f_in);
io_flush();
report(f_out);
***
*** 421,427
close(error_pipe[1]);
if (f_in != f_out) close(f_in);
! io_start_buffering(f_out);
io_set_error_fd(error_pipe[0]);
--- 423,429
close(error_pipe[1]);
if (f_in != f_out) close(f_in);
! io_start_buffering_out(f_out);
io_set_error_fd(error_pipe[0]);
***
*** 434,440
write_int(f_out, -1);
}
io_flush();
!
kill(pid, SIGUSR2);
wait_process(pid, status);
return status;
--- 436,442
write_int(f_out, -1);
}
io_flush();
! msleep(100);
kill(pid, SIGUSR2);
wait_process(pid, status);
return status;
***
*** 476,481
--- 478,484
}
}
+ io_start_buffering_in(f_in);
if (delete_mode !delete_excluded)
recv_exclude_list(f_in);
***
*** 569,574
--- 572,578
extern int cvs_exclude;
extern int delete_mode;
extern int delete_excluded;
+ io_start_buffering_out(f_out);
if (cvs_exclude)
add_cvs_excludes();
if (delete_mode !delete_excluded)
***
*** 578,584
--- 582,591
if (verbose 3)
rprintf(FINFO,file list sent\n);
+ io_flush();
+ io_start_buffering_out(f_out);
send_files(flist,f_out,f_in);
+ io_flush();
if (remote_version = 24) {
/* final goodbye message */
read_int(f_in);
***
*** 590,595
--- 597,603
wait_process(pid, status);
}
report(-1);
+ io_flush();
exit_cleanup(status);
}
Mike Rubel wrote:
[ ... ]
Could you clarify for me whether the patch Steve refers to is the same as
(or has the same effect as) the one you describe below? Or are they two
different animals? If they are different, should we test whether this
patch fixes his specific problem?
Best regards,
Mike
http://www.mikerubel.org
--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Bug reporting: http://cygwin.com/bugs.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
