Faster cygwin rsync ssh
I was wondering if there were any flags, patches, etc, that can speed up rsync over SSH between two Win2k boxes running cygwin? I am setting up a system where these machines can sync off each other, and they potentially might transfer up to a Gig of data, so a little bit more speed goes a long way. These machines will be on the same local network. In my tests I have both plugged into the same 10/100 switch and file transfers over SMB are much much faster than over rsync with ssh. I would expect some slowdown using rsync and ssh due to increased overhead and the way cygwin deals with pipes, etc, but I would like the speed to be better. I don't want to drop ssh either. Any advice would be great. Thanks. -- Joseph Annino Consulting - Perl, PHP, MySQL, Oracle, etc. [EMAIL PROTECTED] - http://www.jannino.com -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Incremental Diffs?
I'm using it to backup files from one computer to another, and it works exactly as I thought it would, except that it seems to be copying entire files over when they've change rather than the differences. What specifically leads you to that conclusion? I have it set to extra verbose, and I've been watching the files transfer over. When I append 2K to a 100MB text file and re-rsync, it's pretty obvious it's transferring 100MB, not 2K + whatever overhead the diff takes up. --- Kim Scarborough http://www.unknown.nu/kim/ --- I know of no man I despise more than Shakespeare; it would be positively a relief to my mind to dig him up and throw stones at him. - George Bernard Shaw --- -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Incremental Diffs?
In message [EMAIL PROTECTED], Kim Scarborough writes: I'm using it to backup files from one computer to another, and it works exactly as I thought it would, except that it seems to be copying entire files over when they've change rather than the differences. What specifically leads you to that conclusion? I have it set to extra verbose, and I've been watching the files transfer over. When I append 2K to a 100MB text file and re-rsync, it's pretty obvious it's transferring 100MB, not 2K + whatever overhead the diff takes up. I wouldn't be so sure. Add the option --stats to the rsync command line and see what it says. AFAIK those numbers are correct. -- Alberto Alberto Accomazzi mailto:[EMAIL PROTECTED] NASA Astrophysics Data System http://adsabs.harvard.edu Harvard-Smithsonian Center for Astrophysicshttp://cfawww.harvard.edu 60 Garden Street, MS 83, Cambridge, MA 02138 USA -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
[PATCH][RFC] space saving incrementals
Please CC me directly as i'm not on the list.
I have attached a patch against latest CVS (cvs diff -u)
that adds the following functionality. I can break it up if
you would prefer it in pieces. Comments welcome.
o add compare-perms option
This creates a new inode for a file even if only
the perms have changed. This way if a file
outside of destdir is hardlinked to a dentry
inside destdir the permissions (uid, gid,
mode) will be untouched preserving history
etc.
o link-dest option
After setting compare_dest this causes
unchanged files in destdir to be hardlinked
to link-dest.
o modified make_exclude_list to support stdin
if --exclude-from has argument of -
stdin will be read.
This lets us pipe a include/exclude list
into rsync so that we can generate it on the
fly.
The upshot of these is to allow rsync to make incremental
backups without modifying earlier versions but keep every
version as a complete tree. It then becomes possible to
make tapes or restore from any image.
Although --compare-perms saves on block count this patch
only applies it to regular files so symlinks and device
nodes will still chew up inodes. :(
for the sake of an example here is the command line i'm using:
/site/bin/rsync -v --stats -a -H -e ssh --compare-perms
--delete --delete-excluded --numeric-ids --exclude-from -
--link-dest /e/backup1/home/update1/tree leto:/efs/home/
/e/backup1/home/update2/tree /e/backup1/home/update2/log
i have tried to keep my changes small and consistent with the
existing coding style. Feel free to dink with it. I only
care about the performance.
--
J.W. SchultzPegasystems Technologies
email address: [EMAIL PROTECTED]
Remember Cernan and Schmitt
Index: exclude.c
===
RCS file: /cvsroot/rsync/exclude.c,v
retrieving revision 1.42
diff -u -r1.42 exclude.c
--- exclude.c 18 Feb 2002 19:10:28 - 1.42
+++ exclude.c 7 Mar 2002 20:56:02 -
@@ -219,8 +219,14 @@
int fatal, int include)
{
struct exclude_struct **list=list1;
- FILE *f = fopen(fname,r);
+ FILE *f;
char line[MAXPATHLEN];
+
+ if (strcmp(fname, -)) {
+ f = fopen(fname,r);
+ } else {
+ f = fdopen(0, r);
+ }
if (!f) {
if (fatal) {
rsyserr(FERROR, errno,
Index: generator.c
===
RCS file: /cvsroot/rsync/generator.c,v
retrieving revision 1.33
diff -u -r1.33 generator.c
--- generator.c 7 Feb 2002 16:36:12 - 1.33
+++ generator.c 7 Mar 2002 20:56:02 -
@@ -42,6 +42,8 @@
extern int always_checksum;
extern int modify_window;
extern char *compare_dest;
+extern int compare_perms;
+extern int link_dest;
/* choose whether to skip a particular file */
@@ -51,6 +53,15 @@
if (st-st_size != file-length) {
return 0;
}
+ if (compare_perms) {
+ if((st-st_mode ~_S_IFMT) != (file-mode ~_S_IFMT)) {
+ return 0;
+ }
+ if (st-st_uid != file-uid || st-st_gid != file-gid) {
+ return 0;
+ }
+ }
+
/* if always checksum is set then we use the checksum instead
of the file time to determine whether to sync */
@@ -352,6 +363,17 @@
statret = -1;
if (statret == -1)
errno = saveerrno;
+#if HAVE_LINK
+ else if (link_dest)
+ if (do_link(fnamecmpbuf, fname) != 0) {
+ if (verbose 0)
+ rprintf(FINFO,link %s = %s : %s\n,
+ fnamecmpbuf,
+ fname,
+ strerror(errno));
+ fnamecmp = fnamecmpbuf;
+ }
+#endif
else
fnamecmp = fnamecmpbuf;
}
Index: options.c
===
RCS file: /cvsroot/rsync/options.c,v
retrieving revision 1.80
diff -u -r1.80 options.c
--- options.c 27 Feb 2002 22:49:57 - 1.80
+++ options.c 7 Mar 2002 20:56:03 -
@@ -106,6 +106,9 @@
int always_checksum = 0;
int list_only = 0;
+int compare_perms = 0;
+int link_dest = 0;
+
char *batch_prefix = NULL;
static int modify_window_set;
@@ -195,6 +198,7 @@
rprintf(F, --safe-linksignore links
Re: Compile 2.5.2 on RedHat 6.2
For what it's worth I saw the same error. 2.5.1 worked fine however. I never found a solution. -- Scott In mailinglists.external.rsync, you wrote: I am running RedHat 6.2 on many of my servers and am receiving the following errors when compiling Rsync 2.5.2: [root@wrapguy rsync-2.5.2]# make gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -c rsync.c -o rsync.o In file included from rsync.c:23: rsync.h:339: warning: no semicolon at end of struct or union rsync.h:339: parse error before `inode' rsync.h:341: parse error before `dev' rsync.h:341: warning: type defaults to `int' in declaration of `dev' rsync.h:341: warning: data definition has no type or storage class rsync.h:344: parse error before `rdev' rsync.h:344: warning: type defaults to `int' in declaration of `rdev' rsync.h:344: warning: data definition has no type or storage class rsync.h:347: `basename' redeclared as different kind of symbol /usr/include/string.h:317: previous declaration of `basename' rsync.h:350: `link' redeclared as different kind of symbol /usr/include/unistd.h:678: previous declaration of `link' rsync.h:352: parse error before `}' rsync.h: In function `flist_up': rsync.h:420: dereferencing pointer to incomplete type rsync.c: In function `set_perms': rsync.c:165: dereferencing pointer to incomplete type rsync.c:168: dereferencing pointer to incomplete type rsync.c:178: dereferencing pointer to incomplete type rsync.c:179: dereferencing pointer to incomplete type rsync.c:179: dereferencing pointer to incomplete type rsync.c:184: dereferencing pointer to incomplete type rsync.c:188: dereferencing pointer to incomplete type rsync.c:189: dereferencing pointer to incomplete type rsync.c:206: dereferencing pointer to incomplete type rsync.c:208: dereferencing pointer to incomplete type rsync.c:152: warning: `change_uid' might be used uninitialized in this function rsync.c:152: warning: `change_gid' might be used uninitialized in this function rsync.c: In function `finish_transfer': rsync.c:245: dereferencing pointer to incomplete type make: *** [rsync.o] Error 1 -- Regards, Scott Russell ([EMAIL PROTECTED]) Linux Technology Center, System Admin, RHCE. T/L 441-9289 / External 919-543-9289 http://bzimage.raleigh.ibm.com/webcam -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
rsync processes never die
Greets. I'm using rsync 2.5.2 but have seen this with 2.5.1. My system is Red Hat 7.2 with the current errata updates. Using rsyncd with xinetd it seems as if old processes never go away. For example right now I have 3 open rsync connections according to netstat -nap but 30 rsync --daemon processes going! All of them show: [root@root]$ ps ax | grep rsync 24950 ?S 0:05 rsync --daemon 25930 ?S 0:31 rsync --daemon ... Any clues as to what's going on? Any way I can get more info about this issue? Nothing is logged into my /var/log/rsyncd.log that is helpful. Oh yeah, the only options in my /etc/rsyncd.conf are timeout 600 and the logging directive. Thanks. -- Regards, Scott Russell ([EMAIL PROTECTED]) Linux Technology Center, System Admin, RHCE. T/L 441-9289 / External 919-543-9289 http://bzimage.raleigh.ibm.com/webcam -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Compile 2.5.2 on RedHat 6.2
OK - that's 3 individuals with Redhat 6.2 installs who have encountered this. I'm pretty sure it's a real problem (ie I can duplicate it and others have had it, too). Is there a more appropriate place to submit this bug or something that I can do to help have it fixed? I am not competent to debug the C code, but I am willing and able to provide any debugging info. Thanks! Matt On Thursday 07 March 2002 03:19 pm, Scott Russell wrote: For what it's worth I saw the same error. 2.5.1 worked fine however. I never found a solution. -- Scott In mailinglists.external.rsync, you wrote: I am running RedHat 6.2 on many of my servers and am receiving the following errors when compiling Rsync 2.5.2: [root@wrapguy rsync-2.5.2]# make gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -c rsync.c -o rsync.o In file included from rsync.c:23: rsync.h:339: warning: no semicolon at end of struct or union rsync.h:339: parse error before `inode' rsync.h:341: parse error before `dev' rsync.h:341: warning: type defaults to `int' in declaration of `dev' rsync.h:341: warning: data definition has no type or storage class rsync.h:344: parse error before `rdev' rsync.h:344: warning: type defaults to `int' in declaration of `rdev' rsync.h:344: warning: data definition has no type or storage class rsync.h:347: `basename' redeclared as different kind of symbol /usr/include/string.h:317: previous declaration of `basename' rsync.h:350: `link' redeclared as different kind of symbol /usr/include/unistd.h:678: previous declaration of `link' rsync.h:352: parse error before `}' rsync.h: In function `flist_up': rsync.h:420: dereferencing pointer to incomplete type rsync.c: In function `set_perms': rsync.c:165: dereferencing pointer to incomplete type rsync.c:168: dereferencing pointer to incomplete type rsync.c:178: dereferencing pointer to incomplete type rsync.c:179: dereferencing pointer to incomplete type rsync.c:179: dereferencing pointer to incomplete type rsync.c:184: dereferencing pointer to incomplete type rsync.c:188: dereferencing pointer to incomplete type rsync.c:189: dereferencing pointer to incomplete type rsync.c:206: dereferencing pointer to incomplete type rsync.c:208: dereferencing pointer to incomplete type rsync.c:152: warning: `change_uid' might be used uninitialized in this function rsync.c:152: warning: `change_gid' might be used uninitialized in this function rsync.c: In function `finish_transfer': rsync.c:245: dereferencing pointer to incomplete type make: *** [rsync.o] Error 1 --- -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: Compile 2.5.2 on RedHat 6.2
OK - that's 3 individuals with Redhat 6.2 installs who have encountered this. I'm pretty sure it's a real problem (ie I can duplicate it and others have had it, too). Is there a more appropriate place to submit this bug or something that I can do to help have it fixed? I am not competent to debug the C code, but I am willing and able to provide any debugging info. Thanks! Matt On Thursday 07 March 2002 03:19 pm, Scott Russell wrote: For what it's worth I saw the same error. 2.5.1 worked fine however. I never found a solution. -- Scott In mailinglists.external.rsync, you wrote: I am running RedHat 6.2 on many of my servers and am receiving the following errors when compiling Rsync 2.5.2: [root@wrapguy rsync-2.5.2]# make gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -c rsync.c -o rsync.o In file included from rsync.c:23: rsync.h:339: warning: no semicolon at end of struct or union rsync.h:339: parse error before `inode' rsync.h:341: parse error before `dev' rsync.h:341: warning: type defaults to `int' in declaration of `dev' rsync.h:341: warning: data definition has no type or storage class rsync.h:344: parse error before `rdev' rsync.h:344: warning: type defaults to `int' in declaration of `rdev' rsync.h:344: warning: data definition has no type or storage class rsync.h:347: `basename' redeclared as different kind of symbol /usr/include/string.h:317: previous declaration of `basename' rsync.h:350: `link' redeclared as different kind of symbol /usr/include/unistd.h:678: previous declaration of `link' rsync.h:352: parse error before `}' rsync.h: In function `flist_up': rsync.h:420: dereferencing pointer to incomplete type rsync.c: In function `set_perms': rsync.c:165: dereferencing pointer to incomplete type rsync.c:168: dereferencing pointer to incomplete type rsync.c:178: dereferencing pointer to incomplete type rsync.c:179: dereferencing pointer to incomplete type rsync.c:179: dereferencing pointer to incomplete type rsync.c:184: dereferencing pointer to incomplete type rsync.c:188: dereferencing pointer to incomplete type rsync.c:189: dereferencing pointer to incomplete type rsync.c:206: dereferencing pointer to incomplete type rsync.c:208: dereferencing pointer to incomplete type rsync.c:152: warning: `change_uid' might be used uninitialized in this function rsync.c:152: warning: `change_gid' might be used uninitialized in this function rsync.c: In function `finish_transfer': rsync.c:245: dereferencing pointer to incomplete type make: *** [rsync.o] Error 1 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
delete fails to delete everything it should like dangling symlinks
I think someone posted this before, but I can't find it in the archives. I am using rsync to pull down source files to be compiled. The delete options are used to clear out any old files left over from previous. Normally this works. I've run into one case where it persistently fails. Within the directory created during compiling is a symlink to another directory, also created during compiling (well, literally, during tar extraction done by the script that does the compiling). When rsync is run to re-syncronize, which should delete all created files (including all those extracted from the tar file), the directory that is the target of the symlink apparently is deleted first. Then when the symlink is encountered, I get an error saying that readlink gets No such file or directory. This doesn't make sense since readlink should work on a dangling link. Here's a paste of manually doing the rsync within the same host (warning, the first line is 318 characters long): = root@pollux:/home/root/src 152 rsync --checksum --copy-unsafe-links --compress '--exclude=*~' '--exclude=#*#' --force --group --links --owner --partial --perms --progress --recursive --timeout=150 --times --verbose --delete --delete-after --delete-excluded --ignore-errors --force /home/root/src-pub/ /home/root/src/ rsync: building file list... rsync: 809 files to consider. readlink openssl-0.9.6b/work/openssl-0.9.6b/crypto/des/asm/perlasm: No such file or directory readlink openssl-0.9.6c/tmp/openssl-0.9.6c/crypto/des/asm/perlasm: No such file or directory wrote 34720 bytes read 20 bytes 4087.06 bytes/sec total size is 85523954 speedup is 2461.83 rsync error: partial transfer (code 23) at main.c(576) root@pollux:/home/root/src 153 readlink openssl-0.9.6c/tmp/openssl-0.9.6c/crypto/des/asm/perlasm ../../perlasm root@pollux:/home/root/src 154 rsync --version rsync version 2.5.2 protocol version 26 Copyright (C) 1996-2002 by Andrew Tridgell and others http://rsync.samba.org/ Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, IPv6, 32-bit system inums, 64-bit internal inums root@pollux:/home/root/src 155 = As you can see I'm using a number of options that should, according to the man page, make sure things get deleted. But this is not happening. I assume that partial transfer (code 23) error message is because it did recognize that it failed to completely get things in sync. BTW, after manually deleting the 2 symlinks, it works fine: = root@pollux:/home/root/src 155 rm -f openssl-0.9.6b/work/openssl-0.9.6b/crypto/des/asm/perlasm root@pollux:/home/root/src 156 rm -f openssl-0.9.6c/tmp/openssl-0.9.6c/crypto/des/asm/perlasm root@pollux:/home/root/src 157 rsync --checksum --copy-unsafe-links --compress '--exclude=*~' '--exclude=#*#' --force --group --links --owner --partial --perms --progress --recursive --timeout=150 --times --verbose --delete --delete-after --delete-excluded --ignore-errors --force /home/root/src-pub/ /home/root/src/ rsync: building file list... rsync: 809 files to consider. deleting directory openssl-0.9.6c/tmp/openssl-0.9.6c/crypto/des/asm deleting directory openssl-0.9.6c/tmp/openssl-0.9.6c/crypto/des deleting directory openssl-0.9.6c/tmp/openssl-0.9.6c/crypto deleting directory openssl-0.9.6c/tmp/openssl-0.9.6c deleting directory openssl-0.9.6c/tmp deleting directory openssl-0.9.6b/work/openssl-0.9.6b/crypto/des/asm deleting directory openssl-0.9.6b/work/openssl-0.9.6b/crypto/des deleting directory openssl-0.9.6b/work/openssl-0.9.6b/crypto deleting directory openssl-0.9.6b/work/openssl-0.9.6b deleting directory openssl-0.9.6b/work wrote 34720 bytes read 20 bytes 4087.06 bytes/sec total size is 85523954 speedup is 2461.83 root@pollux:/home/root/src 158 = -- - | Phil Howard - KA9WGN | Dallas | http://linuxhomepage.com/ | | [EMAIL PROTECTED] | Texas, USA | http://phil.ipal.org/ | - -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
