Re: Java client and rsync
Holstein, Brian wrote: Are there any available java classes that can use rsync for file transfer. Ifso do they support connecting to an rsync-server. Any reasons to or notto do this from within java would also be appreciated. Andre John Mas was trying to create a java rsync library and I was trying to help him but it seems that the both of us have too little spare time to get things going... https://sourceforge.net/projects/jrsync/ -- Lapo 'Raist' Luchini [EMAIL PROTECTED] (PGP X.509 keys available) http://www.lapo.it (ICQ UIN: 529796)
unexpected EOF in read_timeout
I've got this when i backup my server ( like 3GB and lots of small files ). i've read this : http://rsync.samba.org/rsync/fom-serve/cache/73.html. i launch rsync like this : rsync -avrpogtzH --rsync-path=/usr/local/bin/rsync --timeout=0 IP_SERVER::cgi-bin /backup and also have error like this : read error: Connection reset by peer unexpected EOF in read_timeout I try rsync 2.4.5, 2.4.6 and 2.5.1 and always the same thing. Both client and server are running linux redhat. I don't know what to do now :( Thanks David
Re: unexpected EOF in read_timeout
Other things : client disk not full : /dev/hdb2 29412000 16321084 11596872 58% /backup I also had RAM to client : 128MB = 256 MB on an intel celeron 600Mhz. This error isn't due to a network error 'cause this happened again and again each night ... The rsync client is running via cron and on the server i connect to a daemon ( rsync --daemon ) error : rsync: read error: Connexion reset by peer rsync error: error in rsync protocol data stream (code 12) at io.c(151) rsync: connection unexpectedly closed (5415230 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(139) and happen when reading a folder on the server which contain such as 70 000 différent files !! I also try to remove the -z but seems not to solve anything ... :( - Original Message - From: David Bordas [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, February 01, 2002 9:38 AM Subject: unexpected EOF in read_timeout I've got this when i backup my server ( like 3GB and lots of small files ). i've read this : http://rsync.samba.org/rsync/fom-serve/cache/73.html. i launch rsync like this : rsync -avrpogtzH --rsync-path=/usr/local/bin/rsync --timeout=0 IP_SERVER::cgi-bin /backup and also have error like this : read error: Connection reset by peer unexpected EOF in read_timeout I try rsync 2.4.5, 2.4.6 and 2.5.1 and always the same thing. Both client and server are running linux redhat. I don't know what to do now :( Thanks David
@ERROR: auth failed on module test
rsync: connection unexpectedly closed (88 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(139) I'm getting the above error when trying to use rsync over port 873. The command I'm using is : rsync -avz root@sundog::test ./ the /etc/rsyncd.conf contains the following auth users = root uid = root gid = staff [test] path = /export/home/mrl315 Any suggestions would be appreciated - I've been trying to get this working off and on for about 2 weeks now. Regards, Mike Lefebvre
Re: @ERROR: auth failed on module test
MICHAEL R. LEFEBVRE schrieb am Fri, Feb 01, 2002 at 09:01:07AM -0500: rsync: connection unexpectedly closed (88 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(139) I'm getting the above error when trying to use rsync over port 873. The command I'm using is : rsync -avz root@sundog::test ./ the /etc/rsyncd.conf contains the following auth users = root uid = root gid = staff [test] path = /export/home/mrl315 Any suggestions would be appreciated - I've been trying to get this working off and on for about 2 weeks now. Server side: - Where is your secrets file? - Does it contain a root:somepassword line? - Does it have the right permissions? Client side: - Did you supply the correct RSYNC_PASSWORD environment prior issuing any rsync commands? General: - Which OS do you use? - Which rsync version do you use? Regards, - Birger
Re: unexpected EOF in read_timeout
That error message can be caused by anything which causes the remote side to go away, so it's not very helpful. The server's log file may have more information. It is possible that you're running out of RAM as is susggested in that FAQ-O-Matic thread; rsync uses up some memory for every file in a run, so if you can split it up into smaller transfers that can help a lot. Watch the available RAM swap space on the server side during a transfer. - Dave Dykstra On Fri, Feb 01, 2002 at 11:15:36AM +0100, David Bordas wrote: Other things : client disk not full : /dev/hdb2 29412000 16321084 11596872 58% /backup I also had RAM to client : 128MB = 256 MB on an intel celeron 600Mhz. This error isn't due to a network error 'cause this happened again and again each night ... The rsync client is running via cron and on the server i connect to a daemon ( rsync --daemon ) error : rsync: read error: Connexion reset by peer rsync error: error in rsync protocol data stream (code 12) at io.c(151) rsync: connection unexpectedly closed (5415230 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(139) and happen when reading a folder on the server which contain such as 70 000 différent files !! I also try to remove the -z but seems not to solve anything ... :( - Original Message - From: David Bordas [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, February 01, 2002 9:38 AM Subject: unexpected EOF in read_timeout I've got this when i backup my server ( like 3GB and lots of small files ). i've read this : http://rsync.samba.org/rsync/fom-serve/cache/73.html. i launch rsync like this : rsync -avrpogtzH --rsync-path=/usr/local/bin/rsync --timeout=0 IP_SERVER::cgi-bin /backup and also have error like this : read error: Connection reset by peer unexpected EOF in read_timeout I try rsync 2.4.5, 2.4.6 and 2.5.1 and always the same thing. Both client and server are running linux redhat. I don't know what to do now :( Thanks David
Re: error in rsync protocol data stream (code 12) at token.c(288)
On Thu, Jan 31, 2002 at 03:26:16PM -0800, Stuart Anderson wrote:
I am getting the following error when mirroring part of the RedHat
distribution tree over a slow connection (~T1 speed). When running
over a faster network (100BaseT) the problem does not appear. Note,
the problem file a large 600MB ISO image, whereas other small files
appear to be fine.
rsync: open connection using /path/ssh remote.host /path/rsync --server -vlHogDtprRz
--timeout=600 --delete --force . /
rsync: building file list...
rsync: 90108 files to consider.
export/mirror/linux/redhat/7.1/en/iso/i386/
/export/mirror/linux/redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso
deflate on token returned 0 (16384 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(288)
rsync finished
The file has in fact been mirrored correctly (md5sum is identical),
however, the file is left with today's date rather synchronizing the
file times as specified.
A repeat of the rsync command with the file already on the destination
machines results in the same error.
This is running between two Sun servers running Solaris 8 and using:
rsync version 2.5.2 protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
http://rsync.samba.org/
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no IPv6,
64-bit system inums, 64-bit internal inums
That's a different symptom than what we've seen before, but have you
applied the following patch that I posted on Tuesday? Rsync 2.5.2 is badly
broken without it.
- Dave Dykstra
--- match.c.O Tue Jan 29 15:31:37 2002
+++ match.c Tue Jan 29 15:31:54 2002
@@ -246,7 +246,7 @@
match. The 3 reads are caused by the
running match, the checksum update and the
literal send. */
- if (offset-last_match = CHUNK_SIZE+s-n
+ if (offset-last_match = CHUNK_SIZE+(int)s-n
(end-offset CHUNK_SIZE)) {
matched(f,s,buf,offset - s-n, -2);
}
Re: @ERROR: auth failed on module test
You need a secrets file as rsyncd.conf.5 says: auth users The auth users option specifies a comma and space separated list of usernames that will be allowed to connect to this module. The usernames do not need to exist on the local system. The usernames may also contain shell wildcard characters. If auth users is set then the client will be challenged to supply a username and password to connect to the module. A challenge response authentication proto- col is used for this exchange. The plain text user- names are passwords are stored in the file speci- fied by the secrets file option. The default is for all users to be able to connect without a pass- word (this is called anonymous rsync). - Dave Dykstra On Fri, Feb 01, 2002 at 09:01:07AM -0500, MICHAEL R. LEFEBVRE wrote: rsync: connection unexpectedly closed (88 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(139) I'm getting the above error when trying to use rsync over port 873. The command I'm using is : rsync -avz root@sundog::test ./ the /etc/rsyncd.conf contains the following auth users = root uid = root gid = staff [test] path = /export/home/mrl315 Any suggestions would be appreciated - I've been trying to get this working off and on for about 2 weeks now. Regards, Mike Lefebvre
Unixware Rsync compilation
Hi! I have some problems compiling rsync.2.5.2 in Unixware 7.1.1. Some examples: popt/poptconfig.c:18: warning: subscript has type `char' options.c:343: warning: missing initializer options.c:343: warning: (near initialization for `long_options[67].val') all of them are warnings, and when I compiling with gmake, the rsync executable apparently runs. Somebody obtains something similar? Thanks a lot, AJ
Re: Fix for batch mode (was Re: batch mode maintainability)
On Thu, Jan 31, 2002 at 02:42:47PM -0801, Jos Backus wrote: ... +Batch mode can be used to apply the same set of updates to many identical +systems\. Suppose one has a directory tree which is replicated on a number of +hosts\. Now suppose some changes have to be made to this source directory tree +and those changes need to be propagated to the other hosts\. In order to do +this using batch mode, the first step is to make a copy of the source +directory tree before the changes are applied; this is called the original +source directory tree\. The changes are then applied to the new source +directory tree\. It doesn't have to be an extra copy, it can be on any of the hosts. Also, I think that calling the two directories the original source directory and source directory is confusing. The usual terminology in the man page is source and destination. How about calling it something like the primary destination tree to indicate that it is first, and say that commonly it is on the same host that the source tree is on. I'd drop directory and just use source tree and primary destination tree. Next, rsync is run with the write-batch option to apply the +changes made to the new source directory tree to the original source directory +tree\. The write-batch option causes the information needed to repeat this +operation against another original source directory tree to be stored in a +batch update fileset (see below) by the rsync client\. The filename of each +file in the fileset starts with a prefix specified by the user as an argument +to the write-batch option\. This fileset is then copied to each remote host, +where rsync is run with the read-batch option, again specifying the same +prefix, and the source directory tree\. There's another source directory tree. That should be destination tree. Rsync updates the source directory +tree using the information stored in the batch update fileset\. +.PP +The fileset consists of 4 files: +.IP o +prefix\fB.rsync_argvs\fP command-line arguments +.IP o +prefix\fB.rsync_flist\fP rsync internal file metadata +.IP o +prefix\fB.rsync_csums\fP rsync checksums +.IP o +prefix\fB.rsync_delta\fP data blocks for file update change +.PP +The .rsync_argvs file contains a command-line suitable for updating a source +directory tree using that batch update fileset\. It can be executed using a +Bourne(-like) shell, optionally passing in an alternate source directory tree +pathname\ which is then used instead of the original path\. This is useful +when the source directory tree path differs from the original source directory +tree path\. +.PP +Generating the batch update fileset once saves having to perform the file +status, checksum and data block generation more than once when updating +multiple source directory trees\. Multicast transport protocols can be used +to transfer the batch update files in parallel to many hosts at once, instead +of sending the same data to every host individually\. +.PP +Example: +.PP +\fBCaveats\fP: +.IP o +The read-batch option expects the source directory tree it is meant to update +to be identical to the source directory tree that was used to create the batch +update fileset\. When a difference between the source directory trees is +encountered the update will fail at that point, leaving the source directory +tree in a partially updated state\. In that case, rsync can be used in its +regular (non-batch) mode of operation to fix up the source directory tree\. Add that the rsync version used on all destinations should be identical to to the one used on the original destination. +.IP o +The -z/--compress option does not work in batch mode and yields a usage +error\. Add that people can instead compress the files with a separate compression tool for transport to the destination. Hmm, I wonder if it would be easy to use rsync's compression library to compress the whole flist, csum, and delta files on the fly. That would certainly be more convenient. +.IP o +The -n/--dryrun option does not work in batch mode and yields a runtime +error\. - Dave Dykstra
Re: error in rsync protocol data stream (code 12) at token.c(288)
According to Dave Dykstra:
On Thu, Jan 31, 2002 at 03:26:16PM -0800, Stuart Anderson wrote:
I am getting the following error when mirroring part of the RedHat
distribution tree over a slow connection (~T1 speed). When running
over a faster network (100BaseT) the problem does not appear. Note,
the problem file a large 600MB ISO image, whereas other small files
appear to be fine.
rsync: open connection using /path/ssh remote.host /path/rsync --server
-vlHogDtprRz --timeout=600 --delete --force . /
rsync: building file list...
rsync: 90108 files to consider.
export/mirror/linux/redhat/7.1/en/iso/i386/
/export/mirror/linux/redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso
deflate on token returned 0 (16384 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(288)
rsync finished
The file has in fact been mirrored correctly (md5sum is identical),
however, the file is left with today's date rather synchronizing the
file times as specified.
A repeat of the rsync command with the file already on the destination
machines results in the same error.
This is running between two Sun servers running Solaris 8 and using:
rsync version 2.5.2 protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
http://rsync.samba.org/
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no IPv6,
64-bit system inums, 64-bit internal inums
That's a different symptom than what we've seen before, but have you
applied the following patch that I posted on Tuesday? Rsync 2.5.2 is badly
broken without it.
- Dave Dykstra
--- match.c.O Tue Jan 29 15:31:37 2002
+++ match.c Tue Jan 29 15:31:54 2002
@@ -246,7 +246,7 @@
match. The 3 reads are caused by the
running match, the checksum update and the
literal send. */
- if (offset-last_match = CHUNK_SIZE+s-n
+ if (offset-last_match = CHUNK_SIZE+(int)s-n
(end-offset CHUNK_SIZE)) {
matched(f,s,buf,offset - s-n, -2);
}
No I did not apply the patch, but I verified the same problem with
rsync-2.4.8, is that sufficient?
I also determined that my comment above about working/not-working on
fast/slow network was misleading. The important discriminant is whether
compression was turned on or not: works without out it, fails with it
(my driver script was automatically selecting compression for slow
networks).
I also found on deja-news reports that this is a known and fixed problem
with zlib-1.1.2. Any chance of rsync upgrading to zlib-1.1.3?
--
Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba
Re: error in rsync protocol data stream (code 12) at token.c(288)
According to sba:
According to Dave Dykstra:
On Thu, Jan 31, 2002 at 03:26:16PM -0800, Stuart Anderson wrote:
I am getting the following error when mirroring part of the RedHat
distribution tree over a slow connection (~T1 speed). When running
over a faster network (100BaseT) the problem does not appear. Note,
the problem file a large 600MB ISO image, whereas other small files
appear to be fine.
rsync: open connection using /path/ssh remote.host /path/rsync --server
-vlHogDtprRz --timeout=600 --delete --force . /
rsync: building file list...
rsync: 90108 files to consider.
export/mirror/linux/redhat/7.1/en/iso/i386/
/export/mirror/linux/redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso
deflate on token returned 0 (16384 bytes left)
rsync error: error in rsync protocol data stream (code 12) at token.c(288)
rsync finished
The file has in fact been mirrored correctly (md5sum is identical),
however, the file is left with today's date rather synchronizing the
file times as specified.
A repeat of the rsync command with the file already on the destination
machines results in the same error.
This is running between two Sun servers running Solaris 8 and using:
rsync version 2.5.2 protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
http://rsync.samba.org/
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, no
IPv6,
64-bit system inums, 64-bit internal inums
That's a different symptom than what we've seen before, but have you
applied the following patch that I posted on Tuesday? Rsync 2.5.2 is badly
broken without it.
- Dave Dykstra
--- match.c.O Tue Jan 29 15:31:37 2002
+++ match.c Tue Jan 29 15:31:54 2002
@@ -246,7 +246,7 @@
match. The 3 reads are caused by the
running match, the checksum update and the
literal send. */
- if (offset-last_match = CHUNK_SIZE+s-n
+ if (offset-last_match = CHUNK_SIZE+(int)s-n
(end-offset CHUNK_SIZE)) {
matched(f,s,buf,offset - s-n, -2);
}
No I did not apply the patch, but I verified the same problem with
rsync-2.4.8, is that sufficient?
I also determined that my comment above about working/not-working on
fast/slow network was misleading. The important discriminant is whether
compression was turned on or not: works without out it, fails with it
(my driver script was automatically selecting compression for slow
networks).
I also found on deja-news reports that this is a known and fixed problem
with zlib-1.1.2. Any chance of rsync upgrading to zlib-1.1.3?
I have verified that the (int) typecast fix to match.c does not solve
this problem.
I have also determined that a good way to reproduce this problem is
to touch (change the timestamp) on some large iso files and then
attempt to re-rsync them with compression turned on, i.e., no need
to re-push the actual file over the network.
--
Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba
Re: error in rsync protocol data stream (code 12) at token.c(288)
On Fri, Feb 01, 2002 at 08:51:17AM -0800, Stuart Anderson wrote: I also found on deja-news reports that this is a known and fixed problem with zlib-1.1.2. Any chance of rsync upgrading to zlib-1.1.3? Jos Backus posted a patch to do this a few days ago. Try applying his patch and reporting back to the list if it solves your problem. -- albert chin ([EMAIL PROTECTED])
Re: Fix for batch mode (was Re: batch mode maintainability)
Thanks Dave for your comments, I have incorporated your feedback in the patch below. Please review. On Fri, Feb 01, 2002 at 09:49:07AM -0600, Dave Dykstra wrote: Hmm, I wonder if it would be easy to use rsync's compression library to compress the whole flist, csum, and delta files on the fly. That would certainly be more convenient. Yes it would. Any ideas on how to go about this? I'm still (re)reading the rsync papers to ge a better idea of the program's code structure. Index: rsync.1 === RCS file: /cvsroot/rsync/rsync.1,v retrieving revision 1.108 diff -u -r1.108 rsync.1 --- rsync.1 29 Jan 2002 21:52:56 - 1.108 +++ rsync.1 1 Feb 2002 19:21:20 - @@ -306,8 +306,8 @@ --log-format=FORMAT log file transfers using specified format --password-file=FILEget password from FILE --bwlimit=KBPS limit I/O bandwidth, KBytes per second - --read-batch=FILE read batch file - --write-batch write batch file + --write-batch=PREFIXwrite batch fileset starting with PREFIX + --read-batch=PREFIX read batch fileset starting with PREFIX -h, --help show this help screen @@ -801,12 +801,14 @@ result is an average transfer rate equalling the specified limit\. A value of zero specifies no limit\. .IP -.IP \fB--read-batch\fP -Apply a previously generated change batch\. -.IP -.IP \fB--write-batch\fP -Generate a set of files that can be transferred -as a batch update\. +.IP \fB--write-batch=PREFIX\fP +Generate a set of files that can be transferred as a batch update\. Each +filename in the set starts with \fBPREFIX\fP. See the BATCH MODE section +for details. +.IP +.IP \fB--read-batch=PREFIX\fP +Apply a previously generated change batch, using the fileset whose filenames +start with \fBPREFIX\fP\. See the BATCH MODE section for details. .IP .PP .SH EXCLUDE PATTERNS @@ -913,29 +915,80 @@ \fBNote:\fP Batch mode should be considered experimental in this version of rsync\. The interface or behaviour may change before it stabilizes\. .PP -The following call generates 4 files that encapsulate the information -for synchronizing the contents of \fBtarget_dir\fP with the updates found in -\fBsrc_dir\fP -.PP +Batch mode can be used to apply the same set of updates to many +identical systems\. Suppose one has a tree which is replicated on a +number of hosts\. Now suppose some changes have been made to this +source tree and those changes need to be propagated to the other +hosts\. In order to do this using batch mode, rsync is run with the +write-batch option to apply the changes made to the source tree to one +of the destination trees\. The write-batch option causes the rsync +client to store the information needed to repeat this operation against +other destination trees in a batch update fileset (see below)\. The +filename of each file in the fileset starts with a prefix specified by +the user as an argument to the write-batch option\. This fileset is +then copied to each remote host, where rsync is run with the read-batch +option, again specifying the same prefix, and the destination tree\. +Rsync updates the destination tree using the information stored in the +batch update fileset\. +.PP +The fileset consists of 4 files: +.IP o +prefix\fB.rsync_argvs\fP command-line arguments +.IP o +prefix\fB.rsync_flist\fP rsync internal file metadata +.IP o +prefix\fB.rsync_csums\fP rsync checksums +.IP o +prefix\fB.rsync_delta\fP data blocks for file update change +.PP +The .rsync_argvs file contains a command-line suitable for updating a +source tree using that batch update fileset\. It can be executed using +a Bourne(-like) shell, optionally passing in an alternate source tree +pathname\ which is then used instead of the original path\. This is +useful when the source tree path differs from the original source tree +path\. +.PP +Generating the batch update fileset once saves having to perform the +file status, checksum and data block generation more than once when +updating multiple source trees\. Multicast transport protocols can be +used to transfer the batch update files in parallel to many hosts at +once, instead of sending the same data to every host individually\. +.PP +Example: +.PP .RS -$ rsync --write-batch [other rsync options here] \e +$ rsync --write-batch=some_update [other rsync options] \e .br -/somewhere/src_dir /somewhere/target_dir +/this/source/dir /this/original/dir .RE +.RS +$ rsync --read-batch=some_update [other rsync options] \e +.br +/that/original/dir +.RE +.PP +\fBCaveats\fP: +.IP o +The read-batch option expects the destination tree it is meant to update +to be identical to the destination tree that was used to create the +batch update fileset\. When a difference between the destination trees +is encountered the update will fail at that point, leaving the +destination tree in a partially updated
Re: error in rsync protocol data stream (code 12) at token.c(288)
According to Albert Chin: On Fri, Feb 01, 2002 at 08:51:17AM -0800, Stuart Anderson wrote: I also found on deja-news reports that this is a known and fixed problem with zlib-1.1.2. Any chance of rsync upgrading to zlib-1.1.3? Jos Backus posted a patch to do this a few days ago. Try applying his patch and reporting back to the list if it solves your problem. Unfortunately, this does not solve my problem, i.e., the following fix in zlib-1.1.3 is not the one that rsync is triggering: - fix an inflate input buffer bug that shows up on rare but persistent occasions (Mark) At this point the workaround is to turn off compression. -- Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba
Re: error in rsync protocol data stream (code 12) at token.c(288)
On Fri, Feb 01, 2002 at 09:09:18AM -0800, Stuart Anderson wrote: According to sba: No I did not apply the patch, but I verified the same problem with rsync-2.4.8, is that sufficient? No, because rsync-2.4.8 include the same security patch that went into 2.5.2 last weekend. If you could try 2.4.6 or 2.5.1 that would tell us if it was some other problem introduced last weekend or not. ... I have verified that the (int) typecast fix to match.c does not solve this problem. I have also determined that a good way to reproduce this problem is to touch (change the timestamp) on some large iso files and then attempt to re-rsync them with compression turned on, i.e., no need to re-push the actual file over the network. Can you post a publicly-available URL for one that shows the symptom? On Fri, Feb 01, 2002 at 11:48:19AM -0800, Stuart Anderson wrote: According to Albert Chin: On Fri, Feb 01, 2002 at 08:51:17AM -0800, Stuart Anderson wrote: I also found on deja-news reports that this is a known and fixed problem with zlib-1.1.2. Any chance of rsync upgrading to zlib-1.1.3? Jos Backus posted a patch to do this a few days ago. Try applying his patch and reporting back to the list if it solves your problem. Be careful with that, because rsync's zlib is modified from the standard. Has somebody verified that the modification is not removed with the upgrade? Unfortunately, this does not solve my problem, i.e., the following fix in zlib-1.1.3 is not the one that rsync is triggering: - fix an inflate input buffer bug that shows up on rare but persistent occasions (Mark) I wonder if it's worth upgrading if nobody has experienced the problem in rsync. - Dave Dykstra
Re: Unixware Rsync compilation
On Fri, Feb 01, 2002 at 04:54:08PM +0100, Antonio José Morán wrote: Hi! I have some problems compiling rsync.2.5.2 in Unixware 7.1.1. Some examples: popt/poptconfig.c:18: warning: subscript has type `char' options.c:343: warning: missing initializer options.c:343: warning: (near initialization for `long_options[67].val') all of them are warnings, and when I compiling with gmake, the rsync executable apparently runs. Somebody obtains something similar? Warnings are normal in this release, now that rsync by default is adding the gcc option -Wall. I think Martin intends to fix most of them but hasn't gotten around to it yet. - Dave Dykstra
Re: Fix for batch mode (was Re: batch mode maintainability)
On Fri, Feb 01, 2002 at 11:23:33AM -0800, Jos Backus wrote: Thanks Dave for your comments, I have incorporated your feedback in the patch below. Please review. On Fri, Feb 01, 2002 at 09:49:07AM -0600, Dave Dykstra wrote: Hmm, I wonder if it would be easy to use rsync's compression library to compress the whole flist, csum, and delta files on the fly. That would certainly be more convenient. Yes it would. Any ideas on how to go about this? I'm still (re)reading the rsync papers to ge a better idea of the program's code structure. I'm not familiar with that area of the code, but I assume it would be straightforward. Another idea just occurred to me: it would be nice to be able to stream all the batch files over the network rather than storing them temporarily on a disk, especially on the destinations during a --read-batch operation. That could significantly decrease both total elapsed time and disk space usage. I think this could even be done with an external program that creates 3 named pipes or unix-domain sockets on each side and multiplexes the three channels together, but it might be better if it were built in. +Batch mode can be used to apply the same set of updates to many +identical systems\. Suppose one has a tree which is replicated on a +number of hosts\. Now suppose some changes have been made to this +source tree and those changes need to be propagated to the other +hosts\. In order to do this using batch mode, rsync is run with the +write-batch option to apply the changes made to the source tree to one +of the destination trees\. The write-batch option causes the rsync +client to store the information needed to repeat this operation against +other destination trees in a batch update fileset (see below)\. The +filename of each file in the fileset starts with a prefix specified by +the user as an argument to the write-batch option\. This fileset is +then copied to each remote host, where rsync is run with the read-batch +option, again specifying the same prefix, and the destination tree\. +Rsync updates the destination tree using the information stored in the +batch update fileset\. Much better. ... +The .rsync_argvs file contains a command-line suitable for updating a +source tree using that batch update fileset\. It can be executed using +a Bourne(-like) shell, optionally passing in an alternate source tree change source to destination +pathname\ which is then used instead of the original path\. This is +useful when the source tree path differs from the original source tree again change source to destination (twice) ... +.IP o +The rsync version used on all destinations should be identical to to the change to to to to - Dave Dykstra
Re: error in rsync protocol data stream (code 12) at token.c(288)
On Fri, Feb 01, 2002 at 02:18:06PM -0600, Dave Dykstra wrote: Be careful with that, because rsync's zlib is modified from the standard. Has somebody verified that the modification is not removed with the upgrade? I was under the erroneous assumption that these fixes had made it back into zlib-1.1.3. I'm now merging the changes made to zlib-1.1.2 by the rsync team back into 1.1.3, and will repost a new diff against the current rsync CVS soon. -- Jos Backus _/ _/_/_/Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/_/ [EMAIL PROTECTED] _/_/ _/_/_/use Std::Disclaimer;
Re: error in rsync protocol data stream (code 12) at token.c(288)
According to Dave Dykstra: On Fri, Feb 01, 2002 at 09:09:18AM -0800, Stuart Anderson wrote: According to sba: No I did not apply the patch, but I verified the same problem with rsync-2.4.8, is that sufficient? No, because rsync-2.4.8 include the same security patch that went into 2.5.2 last weekend. If you could try 2.4.6 or 2.5.1 that would tell us if it was some other problem introduced last weekend or not. The problem was introduced in version 2.4.4 and exists in all later versions up to and including 2.5.2. Whereas version 2.4.3 and 2.4.0 are both fine (the only earlier versions checked). The problem only exists for compression and large files. I encountered and tested the various rsync versions with the file, redhat/7.1/en/iso/i386/seawolf-i386-powertools.iso gotten from your nearest RedHat mirror, but it is more general than just this file. However, the problem is not reproducible with any of the 1018 small files from redhat/7.1/en/os/i386/RedHat/RPMS which range in size from 1810 Byte to 21541821 Byte. To reproduce: 1) rsync --archive a large file (ISO size?) to a remote machine 2) /bin/touch it on the destination machine 3) rsync --archive --compress again 4) Check for error message: deflate on token returned 0 (16384 bytes left) I don't know if it matters but I am running rsync between two Solaris 8 machines, and using ssh as the transport: ssh -V OpenSSH_2.9p2, SSH protocols 1.5/2.0, OpenSSL 0x0090601f -- Stuart Anderson [EMAIL PROTECTED] http://www.srl.caltech.edu/personnel/sba
Re: Fix for batch mode (was Re: batch mode maintainability)
On Fri, Feb 01, 2002 at 02:59:50PM -0600, Dave Dykstra wrote: I'm not familiar with that area of the code, but I assume it would be straightforward. Good to hear that :) Another idea just occurred to me: it would be nice to be able to stream all the batch files over the network rather than storing them temporarily on a disk, especially on the destinations during a --read-batch operation. That could significantly decrease both total elapsed time and disk space usage. I think this could even be done with an external program that creates 3 named pipes or unix-domain sockets on each side and multiplexes the three channels together, but it might be better if it were built in. Sounds like a cool idea... Again, thanks for the review, technical writing is not one of my forte's. Next installment of the batch mode doc patch: Index: rsync.1 === RCS file: /cvsroot/rsync/rsync.1,v retrieving revision 1.108 diff -u -r1.108 rsync.1 --- rsync.1 29 Jan 2002 21:52:56 - 1.108 +++ rsync.1 1 Feb 2002 23:50:56 - @@ -306,8 +306,8 @@ --log-format=FORMAT log file transfers using specified format --password-file=FILEget password from FILE --bwlimit=KBPS limit I/O bandwidth, KBytes per second - --read-batch=FILE read batch file - --write-batch write batch file + --write-batch=PREFIXwrite batch fileset starting with PREFIX + --read-batch=PREFIX read batch fileset starting with PREFIX -h, --help show this help screen @@ -801,12 +801,14 @@ result is an average transfer rate equalling the specified limit\. A value of zero specifies no limit\. .IP -.IP \fB--read-batch\fP -Apply a previously generated change batch\. -.IP -.IP \fB--write-batch\fP -Generate a set of files that can be transferred -as a batch update\. +.IP \fB--write-batch=PREFIX\fP +Generate a set of files that can be transferred as a batch update\. Each +filename in the set starts with \fBPREFIX\fP. See the BATCH MODE section +for details. +.IP +.IP \fB--read-batch=PREFIX\fP +Apply a previously generated change batch, using the fileset whose filenames +start with \fBPREFIX\fP\. See the BATCH MODE section for details. .IP .PP .SH EXCLUDE PATTERNS @@ -913,29 +915,80 @@ \fBNote:\fP Batch mode should be considered experimental in this version of rsync\. The interface or behaviour may change before it stabilizes\. .PP -The following call generates 4 files that encapsulate the information -for synchronizing the contents of \fBtarget_dir\fP with the updates found in -\fBsrc_dir\fP -.PP +Batch mode can be used to apply the same set of updates to many +identical systems\. Suppose one has a tree which is replicated on a +number of hosts\. Now suppose some changes have been made to this +source tree and those changes need to be propagated to the other +hosts\. In order to do this using batch mode, rsync is run with the +write-batch option to apply the changes made to the source tree to one +of the destination trees\. The write-batch option causes the rsync +client to store the information needed to repeat this operation against +other destination trees in a batch update fileset (see below)\. The +filename of each file in the fileset starts with a prefix specified by +the user as an argument to the write-batch option\. This fileset is +then copied to each remote host, where rsync is run with the read-batch +option, again specifying the same prefix, and the destination tree\. +Rsync updates the destination tree using the information stored in the +batch update fileset\. +.PP +The fileset consists of 4 files: +.IP o +prefix\fB.rsync_argvs\fP command-line arguments +.IP o +prefix\fB.rsync_flist\fP rsync internal file metadata +.IP o +prefix\fB.rsync_csums\fP rsync checksums +.IP o +prefix\fB.rsync_delta\fP data blocks for file update change +.PP +The .rsync_argvs file contains a command-line suitable for updating a +destination tree using that batch update fileset\. It can be executed using a +Bourne(-like) shell, optionally passing in an alternate destination tree +pathname\ which is then used instead of the original path\. This is useful +when the destination tree path differs from the original destination tree +path\. +.PP +Generating the batch update fileset once saves having to perform the file +status, checksum and data block generation more than once when updating +multiple destination trees\. Multicast transport protocols can be used to +transfer the batch update files in parallel to many hosts at once, instead of +sending the same data to every host individually\. +.PP +Example: +.PP .RS -$ rsync --write-batch [other rsync options here] \e +$ rsync --write-batch=some_update [other rsync options] \e .br -/somewhere/src_dir /somewhere/target_dir +/this/source/dir /this/original/dir .RE +.RS +$ rsync --read-batch=some_update [other rsync options]
rsync-2.5.2 possible buglets
Hello, I don't know if this amounts to much, but did you intend to use a rather than a at line 739 of flist.c? Also, lines 101 279 of io.c have a shift right of a signed value. If they could never ever be a negative number, the variable might ought to be change to unsigned. Cheers, Steve Grubb __ Do You Yahoo!? Great stuff seeking new owners in Yahoo! Auctions! http://auctions.yahoo.com
Re: rsync-2.5.2 possible buglets
On Fri, 1 Feb 2002, Steve G wrote: I don't know if this amounts to much, but did you intend to use a rather than a at line 739 of flist.c? Fortunately both items in the expression can only have the value of 1 or 0, so the effect is the same as . It looks like a typo to me, though. ..wayne..
