Identical files showing in dry-run
Hi all, I've got a problem with rsync 2.6.2 on Solaris/SPARC (one system is Solaris 8, the other is Solaris 9). I'm trying to get a list of files that have changed on one server (a development web server) and another (a live web server). I am using rsync as follows :- rsync -Icrn --exclude='.svn' --stats -e ssh [source dir] [EMAIL PROTECTED]:dest dir] As far as I understand, this should stop rsync from using quick checks such as file size and modified time (-I), and forces it to look at the checksums of each file (-c), as well as missing out any Subversion directories (.svn). However, each time I try this, instead of showing only the files that have changed, it returns a list of every file in the source directory. Most of these files are identical on both servers - a check using gmd5sum on both systems confirms this. The same thing happens when I use different criteria, such as --size-only. Any ideas what I'm doing wrong, or what I should be doing ? Thanks in advance, -Mark -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
odd issue.
This is a debian machine. #: /usr/bin/rsync -av -e /usr/bin/ssh /usr/box1backup-072104.tar.gz [EMAIL PROTECTED]:/root/box1backup-072104.tar.gz bash: rsync: command not found rsync: connection unexpectedly closed (0 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) that exact same command works on a redhat box. Why is bashing saying that it cant find rsync even though rsync is then executed? Im pretty confused. -Drew -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Files to short after transfer
High, I am experiencing a problem with rsync. The symptom is that rsync does not seem to notice that the file at the target side is shorter then the source one and then does not update the file. This leaves me with corrupted files at the target side. Now, this does not happen all the time but only for few files. My environment: - rsync 2.6.2 on Tru64 unix 5.1 - I am copying from a local file system to an AFS file system. - My command line: rsync -ruzv /home/TRICS/data/2004/ \ /afs/psi.ch/project/sinqdata/2004/trics/ - The file size is several MB. The files are frequently appended to by our instrument control software. There may be a reading/writing synchronisation problem. - If I delete the file at the target end, a rerun of rsync copies it nicely. As this does not happen very often this must be quite difficult to track down. Any ideas how to get this sorted? I think rsync should notice that files are of different size and copy them then. Best Regards, Mark Koennecke -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Identical files showing in dry-run
On Thu, Jul 22, 2004 at 11:22:10AM +0100, Mark Round wrote: rsync -Icrn --exclude='.svn' --stats -e ssh [source dir] [EMAIL PROTECTED]:dest dir] Any ideas what I'm doing wrong, or what I should be doing ? What you should do is to drop the -I option and just use -c. The -I option tells rsync to transfer everything. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: backup complete server
primary = 10.28.123.232 backup = 10.28.123.230 On the primary server, I have edited my rsyncd.conf file to the following: uid = 0 gid = 0 use chroot = no max connections = 4 syslog facility = local5 pid file = /var/run/rsyncd.pid read only = true list = false hosts allow = 10.28.123.230 [etc] path = /etc comment = system config files [web] path = /var/www comment = web files Now, from the backup server, I run the following command: rsync 10.28.123.232:/var/www -avz /var/www or rsync 10.28.123.232:/var/www -v /var/www And get the following errors: connect to address 10.28.123.232: Connection refused Trying krb4 rsh... connect to address 10.28.123.232: Connection refused trying normal rsh (/usr/bin/rsh) 10.28.123.232: Connection refused rsync: connection unexpectedly closed (0 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(165) Does rsync log anything anywhere so that i can see what's going on, or are there any special debug commands I can use to see where I'm going wrong? Also, when I do a: rsync 10.28.123.232:: -v from the backup server, it proudces nothing - no output at all. Thanks, Scott - Original Message - From: Jim Salter [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, July 19, 2004 12:50 PM Subject: Re: backup complete server Remove the IP address you have stuck in front of the double-colon. rsync -v 10.28.123.232:: That should provide you with a list of the modules you have set up on your rsync daemon on the server. Except that it doesn't look like you have any defined. Here's an example module definition: [etc] path = /etc comment = system config files or if you prefer to give any random rsync user access to everything on the entire server (NOT recommended, particularly if you do not set read-only or auth users / secrets): [root] path = / comment = hope you know what you're doing Hope that helps. Jim Salter JRS Systems I'm new to rsync, and am setting up two servers (Fedora Core 1) - one bing the primary and one the backup. primary: 10.28.123.232 backup: 10.28.123.230 On the primary server, I have the following rsyncd.conf file: pid file = /var/run/rsync.pid path = / use chroot = false read only = true list = false uid = 0 gid = 0 hosts allow = 10.28.123.230 From the backup server, I try to run the following commands, but get errors: rsync -v 10.28.123.232::10.28.123.230 rsync failed to connect to 10.28.123.232: connection refused rsync error: error in socket IO (code 10) at clientserver.c(83) I then tried (from the backup server) the following: rsync -v 10.28.123.232:10.28.123.230 connecting to address 10.28.123.232: Connection refused trying krb4.rsh... connecting to address 10.28.123.232: Connection refused trying normal rsh (/usr/bin/rsh) 10.28.123.232: Connection refused rsync: connection unexpectedly closed (o bytes read so far) rsync error: error in rsync protocol data stream (code 21) at IO.c (165) From everything i've read, it should be correct, but still not working. Can anyone please point out what I could be doing wrong? My goal is to have the backup server take a complete backup of the /etc and /home and /var directories every night, but want to get at least some files moving before I start specifying what to and not to move. Thanks, Scott -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Incoming mail is certified Virus Free. Checked by AVG Anti-Virus (http://www.grisoft.com). Version: 7.0.253 / Virus Database: 263.4.6 - Release Date: 7/19/2004 -- Outgoing mail is certified Virus Free. Checked by AVG Anti-Virus (http://www.grisoft.com). Version: 7.0.253 / Virus Database: 263.4.8 - Release Date: 7/22/2004 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: odd issue.
On Thu, Jul 22, 2004 at 10:26:45AM -0400, Drew Weaver wrote: Why is bashing saying that it cant find rsync even though rsync is then executed? The remote bash is saying that. You can either fix the PATH on the remote box (e.g. make it so that ssh -l root box2 rsync outputs the rsync help text) or supply the --rsync-path=/usr/local/bin/rsync (or whatever it should be) to rsync. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Files to short after transfer
On Thu, Jul 22, 2004 at 04:03:26PM +0200, Mark Koennecke wrote: - My command line: rsync -ruzv /home/TRICS/data/2004/ \ /afs/psi.ch/project/sinqdata/2004/trics/ The -u option tells rsync to ONLY update the file if the timestamp is earlier than the timestamp of the sending file. This will override the normal size/date check, which is why rsync isn't updating the file. How it got corrupted (most likely) is that the file changed during the transfer and rsync put the result of the transfer into place on the destination pending a resend. (This broken behavior was fixed in the CVS version a while back so that (a) we discard a file that failed to verify unless --partial was specified, and (b) we don't set the timestamp on a file that failed to verify when we do keep it.) Are you using the -u option for some other reason? If not, I'd suggest dropping it. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: multiple rsync simlataneously
On Wed, Jul 21, 2004 at 07:51:56PM +0530, [EMAIL PROTECTED] wrote: Please provide a good solution. The best solution is to not backup 100 hosts every minute. You'll probably have to stagger the hosts so that they don't all try to backup at the same time. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
O_NOATIME in linux 2.6.8rc2
linux 2.6.8rc2 gained an O_NOATIME open option, which allows you to make backups without affecting the atime. Someone might like to add this into rsync as an option. It should be pretty trivial. -- Martin pgpOjwszIPU8F.pgp Description: PGP signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync 2.6.2
On Wed, Jul 21, 2004 at 02:21:27PM -0400, Don Malloy wrote: Hopefully this is of more help to you. Unfortunately I haven't been able to get rsync to fail in the update of a large file, and the system call trace you provided didn't tell me anything helpful in narrowing down what is going wrong. Could you attach a debugger to the halting rsync while it is running, tell it to continue, and get a backtrace of where it dies? (e.g. gdb rsync 1234 where the 1234 is the process id, set a breakpoint on exit_cleanup, tell it to continue and note the output of bt). ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: not sure how to read the log
On Thu, Jul 22, 2004 at 12:45:01AM +0200, Iavor Raytchev wrote: rename LOCAL_PATH_TO_E_MAIL_MESSAGE HERE - REMOTH_PATH_TO_E_MAIL_MESSAGE_HERE: No such file or directory SAME_REMOTHE_PATH_TO_E_MAIL_MESSAGE_HERE That looks like the message from the end of the transfer, but it's hard to know for sure with it mangled like that. The no such file or directory error for a rename typically means that a directory is missing. Or perhaps the filename contains a character that is confusing XP? ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Identical files showing in dry-run
What you should do is to drop the -I option and just use -c. The -I option tells rsync to transfer everything. This doesn't have any change - it still transfers everything :( -Mark --- This E-mail is from IPC Media Ltd whose registered office is at Kings Reach Tower, Stamford Street, London SE1 9LS, registered number 53626. The contents and any attachments to it include information that is private and confidential and should only be read by those persons to whom they are addressed. IPC Media accepts no liability for any loss or damage suffered by any person arising from the use of this e-mail. Neither IPC Media nor the sender accepts any responsibility for viruses and it is your responsibility to check the email and attachments (if any). No contracts may be concluded on behalf of IPC Media by means of e-mail communications. If you have received this e-mail in error, please destroy and delete the message from your computer. For unbeatable savings on magazine subscriptions and great gift ideas visit www.ipcsubs.co.uk/IZAF -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Strange behaviour rsync pull from sun to ppc linux, Integer overflow - attempted 64 bit offset
Hi,
I am running rsync on an embedded linux (ppc) like this:
rsync -avvv sun::file-system/ /rsyncexperiment
Where sun is a solaris machine that has rsync:
rsync version 2.5.7 protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
http://rsync.samba.org/
Capabilities: 64-bit files, socketpairs, hard links, symlinks,
batchfiles, no IPv6, 64-bit system inums, 64-bit internal inums
The ppc machine has rsync, the same version but there is a warning:
rsync version 2.5.7 protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
http://rsync.samba.org/
Capabilities: 64-bit files, no socketpairs, hard links, symlinks,
batchfiles,
IPv6, 64-bit system inums, 64-bit internal inums
WARNING: no 64-bit integers on this platform!
I have read the comment in rsync.h about suns and non 64 bit machines
(maybe I haven't fully absorbed the meaning of it). I have files on the
sun including devices but I'm not concerned about truncation when they
are transferred to the embedded linux.
The 'strange behaviour' is that the transfer completes and the sender
seems to send a -1 (as seen in main.c)
if (remote_version = 24) {
/* send a final goodbye message */
write_int(f_out, -1);
}
The reader reads this and gives an error:
[...]
recv_generator(var/spool/cron/crontabs,1866)
recv_generator(var/spool/repackage,1867)
recv_generator(var/state,1868)
recv_generator(var/tmp,1869)
recv_generator(var/ucd-snmp,1870)
recv_files finished
Integer overflow - attempted 64 bit offset
rsync error: requested action not supported (code 4) at ../io.c(381)
_exit_cleanup(code=4, file=../io.c, line=381): about to call exit(4)
rsync: connection unexpectedly closed (35060 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at ../io.c(165)
_exit_cleanup(code=12, file=../io.c, line=165): about to call exit(12)
If I initiate the transfer from the sun, it completes successfully.
So, is this a bug or expected behaviour?
-Cam
PS, in case it makes a difference, on the sun rsync is configured to run
from inetd, and on the embedded machine it is run in daemon mode by hand.
--
[EMAIL PROTECTED] --
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Strange behaviour rsync pull from sun to ppc linux, Integer overflow - attempted 64 bit offset
On Thu, Jul 22, 2004 at 07:09:03PM +0100, Cam wrote: The 'strange behaviour' is that the transfer completes and the sender seems to send a -1 (as seen in main.c) Not quite. The cause must come from someone calling either read_longint() or write_longint(), which can happen at the end of the transfer in the report() function (if you're doing a pull rather than a push). That is undoubtedly where the error you saw was coming from. Capabilities: 64-bit files, no socketpairs, hard links, symlinks, batchfiles, IPv6, 64-bit system inums, 64-bit internal inums WARNING: no 64-bit integers on this platform! That's a weird contradiction there. I note that if the code in rsync.h can't find a 64-bit type, it falls back to using off_t. However, if off_t is actually 64-bits (as it was here), the code doesn't notice that. I've just checked in some changes into CVS that should take care of this case. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync 2.6.2
I started rsync as follows: server2 # ./rsync -a --progress --stats file1.db /var/opt/ignite building file list ... 1 file to consider file1.db 1951039488 45%4.40MB/s0:08:420:14:56 rsync: writefd_unbuffered failed to write 32768 bytes: phase unknown: Broken pipe (32) rsync error: error in rsync protocol data stream (code 12) at io.c(900) There are always 3 rsync processes running during a sync. Attaching to the first one stopped the transfer after a breakpoint was set. Attaching to the second one yeilded more results. server2 # ./gdb ./rsync 14039 HP gdb 2.1 Copyright 1986 - 1999 Free Software Foundation, Inc. Hewlett-Packard Wildebeest 2.1 (based on GDB 5.0-hpwdb-2630) Wildebeest is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for Wildebeest. Type show warranty for details. Wildebeest was built for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00. .. /archive1/14039: No such file or directory. Attaching to program: /archive1/./rsync, process 14039 Reading symbols from /usr/lib/libc.2...done. Reading symbols from /usr/lib/libdld.2...done. warning: reading `r3' register: No data 0xc020ccd0 in _select_sys () from /usr/lib/libc.2 (gdb) break exit_cleanup Breakpoint 1 (deferred) at exit_cleanup (exit_cleanup was not found). Breakpoint deferred until a shared library containing exit_cleanup is loaded. (gdb) bt #0 0xc020ccd0 in _select_sys () from /usr/lib/libc.2 #1 0xc0218198 in select () from /usr/lib/libc.2 #2 0x2341c in writefd_unbuffered (fd=4, buf=0x40083238 , len=32768) at io.c:863 #3 0x2398c in writefd (fd=4, buf=0x40083238 , len=32768) at io.c:972 #4 0x23d58 in write_buf (f=4, buf=0x40083238 , len=32768) at io.c:1034 #5 0x25fb0 in simple_send_token (f=4, token=-2, buf=0x4000af10, offset=136511488, n=32768) at token.c:104 #6 0x26db4 in send_token (f=4, token=-2, buf=0x4000af10, offset=136511488, n=32768, toklen=0) at token.c:472 #7 0x152c0 in matched (f=4, s=0x4000aee8, buf=0x4000af10, offset=136544256, i=-2) at match.c:113 #8 0x16248 in match_sums (f=4, s=0x4000aee8, buf=0x4000af10, len=4304863232) at match.c:345 #9 0xc280 in send_files (flist=0x4000bef0, f_out=4, f_in=5) at sender.c:232 #10 0x12b10 in client_run (f_in=5, f_out=4, pid=14040, argc=1, argv=0x4000aa00) at main.c:666 #11 0x136d0 in start_client (argc=1, argv=0x4000aa00) at main.c:924 #12 0x13c74 in main (argc=2, argv=0x4000aa00) at main.c:1110 (gdb) quit The program is running. Quit anyway (and detach it)? (y or n) y Detaching from program: /archive1/./rsync, process 14039 server2 # ps -ef | grep rsync root 14040 14039 0 16:16:10 pts/td0:00 ./rsync -a --progress --stats file1.db /var/opt/ignite root 14041 14040 123 16:16:10 pts/td0:17 ./rsync -a --progress --stats file1.db /var/opt/ignite root 14039 13021 121 16:16:10 pts/td0:17 ./rsync -a --progress --stats file1.db /var/opt/ignite server2 # ./gdb ./rsync 14040 HP gdb 2.1 Copyright 1986 - 1999 Free Software Foundation, Inc. Hewlett-Packard Wildebeest 2.1 (based on GDB 5.0-hpwdb-2630) Wildebeest is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for Wildebeest. Type show warranty for details. Wildebeest was built for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00. .. /archive1/14040: No such file or directory. Attaching to program: /archive1/./rsync, process 14040 Reading symbols from /usr/lib/libc.2...done. Reading symbols from /usr/lib/libdld.2...done. warning: reading `r3' register: No data 0xc020ccd0 in _select_sys () from /usr/lib/libc.2 (gdb) break exit_cleanup Breakpoint 1 (deferred) at exit_cleanup (exit_cleanup was not found). Breakpoint deferred until a shared library containing exit_cleanup is loaded. (gdb) bt #0 0xc020ccd0 in _select_sys () from /usr/lib/libc.2 #1 0xc0218198 in select () from /usr/lib/libc.2 #2 0x2202c in read_timeout (fd=4, buf=0x7f7f1d00 , len=4) at io.c:421 #3 0x22984 in read_loop (fd=4, buf=0x7f7f1d00 , len=4) at io.c:619 #4 0x217f0 in read_msg_fd () at io.c:220 #5 0x21c50 in get_redo_num () at io.c:308 #6 0x8c64 in generate_files (f_out=1, flist=0x4000bef0, local_name=0x0) at generator.c:588 #7 0x123d8 in do_recv (f_in=0, f_out=1, flist=0x4000bef0, local_name=0x0) at main.c:502 #8 0x12738 in do_server_recv (f_in=0, f_out=1, argc=1, argv=0x7f7f06d4) at main.c:578 #9 0x12898 in start_server (f_in=0, f_out=1, argc=2, argv=0x7f7f06d0) at main.c:609 #10 0x12790 in child_main (argc=2, argv=0x7f7f06d0) at main.c:585 #11 0x2e0c8 in local_child (argc=2, argv=0x7f7f06d0, f_in=0x7f7f0668, f_out=0x7f7f066c, child_main=0x40004df2 child_main) at pipe.c:153 #12 0x11b7c in do_cmd (cmd=0x0, machine=0x0, user=0x0, path=0x4000aed0 /var/opt/ignite,
ERROR: out of memory in receive_file_entry
Hello, I'm looking for some possible solutions to the out of memory problem when dealing with very large directory trees. Client: linux-2.4.20 Server: HP-UX 11.11 rsync version: 2.6.2 Directory size: 400Gbytes number of files: 3273133 rsync cmd: rsync -avRx --progress --stats --numeric-ids --blocking-io --delete -e ssh hp-ux.server:/path /local/linux/ It seems to fail after consuming roughly 250M of memory and recieving a little over 3mil files in the file list. Both client and server have 1G of RAM and 2G of swap. Here is the error message from rsync: snip ERROR: out of memory in receive_file_entry rsync error: error allocating core memory buffers (code 22) at util.c(116) rsync: connection unexpectedly closed (63453387 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(342) /snip It's hard to tell which system is running out of memory in this case. I'd thought about trying 64bit-ness (we will have an AMD64 linux test box pretty soon), but we are only talking about 250M so I can't imagine that would help at all. I might try it anyways when the box gets here. The only real solution that I have been able to come up with is to divide and conquer. Running multiple rsync processes for this single filesystem so rsync doesn't have to deal with 3mil+ files will succeed. But, 400G is a small filesystem in our environment and this solution will be very difficult to implement on larger filesystems. Any thoughts? -- James Bagley| CDI Innovantage [EMAIL PROTECTED] | Technical Computing UNIX Admin Support DON'T PANIC | Agilent Technologies IT Phone: (541) 738-3340 | Corvallis, Oregon -- -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HP-UX 11i and largefiles on rsync 2.6.2
On Thu, Jul 22, 2004 at 04:27:57PM -0400, Don Malloy wrote: server2 # ./rsync -a --progress --stats file1.db /var/opt/ignite What I want is the backtrace when the rsync that is failing dies so we can (hopefully) figure out why it is dying. In the above command the client should be the sender, so it appears to be receiving side that is dying. Unfortunately that means that there are two processes to choose from (the two higher-numbered ones). The best thing to do would be to attach a debugger to both of them (each in a separate window). Set a break-point at _exit_cleanup (I forgot that the function is really named with a leading underscore because of some macro shenanigans), and then tell it to continue. You'll be able to get a backtrace of what each program was doing when it either died from a signal or chose to exit. You can look around a bit too (e.g. move up, look around, and print some variables). ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Batch-mode rewrite
Wayne, I took a crack a batch-mode test case, to try avoid batch-mode regressions. Does this look reasonable? Also, I came across a confusing typo (I think.) Also attached. -chris Index: testsuite/README.testsuite === RCS file: /cvsroot/rsync/testsuite/README.testsuite,v retrieving revision 1.2 diff -u -r1.2 README.testsuite --- testsuite/README.testsuite 12 Mar 2002 00:22:22 - 1.2 +++ testsuite/README.testsuite 23 Jul 2004 00:54:58 - @@ -17,8 +17,8 @@ path. If the tests pass, you should see a report to that effect. Some tests -require being root or some other precondition, and so will normally be -checked -- look at the test scripts for more information. +require being root or some other precondition, and so will normally not +be checked -- look at the test scripts for more information. If the tests fail, you will see rather more output. The scratch directory will remain in the build directory. It would be useful if #! /bin/sh # Copyright (C) 2004 by Chris Shoemaker [EMAIL PROTECTED] # This program is distributable under the terms of the GNU GPL (see # COPYING). # Test rsync's --write-batch and --read-batch options . $suitedir/rsync.fns set -x hands_setup || test_fail failed to build test directories runtest local --write-batch 'checkit $RSYNC -av --write-batch=BATCH \$fromdir/\ \$todir\ $fromdir/ $todir' rm -rf $todir || test_fail failed to remove destination directory runtest --read-batch 'checkit $RSYNC -av --read-batch=BATCH \$todir\ $fromdir/ $todir' rm -rf $todir || test_fail failed to remove destination directory build_rsyncd_conf RSYNC_CONNECT_PROG=$RSYNC --config=$conf --daemon export RSYNC_CONNECT_PROG runtest daemon sender --write-batch $RSYNC -av --write-batch=BATCH rsync://localhost/test-from/ \$todir\ rm -rf tmptodir mv $todir tmptodir || test_fail failed to save copy of destination directory runtest --read-batch from daemon 'checkit $RSYNC -av --read-batch=BATCH \$todir\ $todir/ tmptodir' rm -rf $todir || test_fail failed to remove destination directory runtest BATCH.sh use of --read-batch 'checkit ./BATCH.sh $todir tmptodir' rm -rf $todir || test_fail failed to remove destination directory mkdir $todir || test_fail failed to restore empty destination directory runtest daemon recv --write-batch 'checkit $RSYNC -av --write-batch=BATCH \$fromdir/\ rsync://localhost/test-to $todir/ tmptodir' if [ x$preserve_scratch != xyes ] then rm -rf tmptodir BATCH BATCH.sh || test_fail failed to remove batch files fi # The script would have aborted on error, so getting here means we pass. exit 0 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Batch-mode rewrite
On Wed, Jul 21, 2004 at 03:54:11PM -0400, Chris Shoemaker wrote: What data exactly? I thought: 1) all recv-to-gen communications went through the error_pipe[] fds. Yes, that became true when I got rid of the extra pipe that used to separate the redo values from the error messages. 2) the only meaningful communications were redo requests and I'm done. It depends on which side is the client. Since we made the read-batch side always the client then the errors/warnings get output to the tty directly (for a client sender, which is the default for non-batch local transfers, the pipe also contains any error messages from the receiver). There is also a patch in the patches dir that sends a success message for every correctly transferred file (delete-sent-files.diff). I thought we could skip the redos and fake the I'm done. Yes, it's possible to turn off the communication, which is what I was talking about -- I was mentioning what things would need to be special-cased. Ah, I see what you mean about the directory tweaks. And don't forget the hard-link post-processing -- it would also need to happen after the receiver finished its job. Just to clarify, I don't have anything against the index notification style gen/recv syncronization. If you think that's better, then let's go that way. In the future there may be a need for data to be communicated back from the generator to the receiver (i.e. if the --fuzzy patch get improved enough to make it worthwhile), so I think in the long run that having the two things running in parallel (like normal) will be the easiest thing to maintain. We can certainly revisit the issue later, but for now I have the code in the generator sending the normal index bytes down the output pipe (it just avoids sending the checksum data), and some special-case read-batch code in the receiver that reads the from-the- generator pipe and only proceeds with an update if the generator said it was time. One interesting thing that fell out of this was the ability to skip any part of a batched update that had already been done. For instance, if you run the ./BATCH.sh file and terminate it (for some reason), you can run it again and it will skip all the already-performed updates. (One exception: if --partial had been specified the interrupted file will probably not update correctly, but rsync should tell you when that happens.) ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Batch-mode rewrite
On Thu, Jul 22, 2004 at 04:10:41PM -0400, Chris Shoemaker wrote: I took a crack a batch-mode test case, to try avoid batch-mode regressions. Does this look reasonable? Yes, that looks good. I made some minor changes and checked it in. Also, I came across a confusing typo (I think.) Also attached. Good catch! Thanks, ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync Performance
On Fri, Jul 23, 2004 at 02:15:04PM +1200, Jason Haar wrote: is there any intention of a new improved --partial option whereby any failed uploads are kept as temp files I had been contemplating whether we need a new option for this or not. One idea would be to change the behavior when --partial was combined with --temp-dir in that the partial file would be renamed to the normal name in the temp dir on abnormal termination and the directory would be preferentially checked for the basis file (kinda like --compare-dest). However, it might be better to create a new option named --partial-dir that would indicate that this new behavior was desired. (For instance, it might surprise someone that the --temp-dir could receive a non-temp format name in it.) Anyone have a preference? Or a better idea? ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync Performance
On Thu, Jul 22, 2004 at 07:33:21PM -0700, Wayne Davison wrote: On Fri, Jul 23, 2004 at 02:15:04PM +1200, Jason Haar wrote: is there any intention of a new improved --partial option whereby any failed uploads are kept as temp files I had been contemplating whether we need a new option for this or not. One idea would be to change the behavior when --partial was combined with --temp-dir in that the partial file would be renamed to the normal name in the temp dir on abnormal termination and the directory would be preferentially checked for the basis file (kinda like --compare-dest). Strange you should say that - I for one assumed that was what would happen if you invoked --partial --temp-dir! I don't know why - but I did. However, it might be better to create a new option named --partial-dir that would indicate that this new behavior was desired. (For instance, it might surprise someone that the --temp-dir could receive a non-temp format name in it.) Yes - that sounds good. It should pretty pretty clean: if you call it as rsync --partial-dir=/path/.rsync-partial/ then it could: 1. create a new subdir, 2. rsync to that, 3. keep files on failure, 4. carry on when called again 5. when finished, move to real location Sanity checks could include: a. deleting temp file if it's date is different than src file (means something weird has happened, so assume it's not there and start from scratch) b. --partial-dir should have to be part of the dst tree - otherwise you potentially have a security risk. e.g. --partial-dir /tmp would mean someone could create /tmp/etc/passwd and you could end up overwriting your /etc/passwd file (if that's what that particular rsync job was replicating of course!). So maybe --partial-path cannot start with ^/? or ^../?. c. on any error, delete and start from scratch again? Keen idea :-) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Batch-mode rewrite
On Thu, Jul 22, 2004 at 06:36:27PM -0700, Wayne Davison wrote:
On Wed, Jul 21, 2004 at 03:54:11PM -0400, Chris Shoemaker wrote:
What data exactly? I thought:
1) all recv-to-gen communications went through the error_pipe[] fds.
Yes, that became true when I got rid of the extra pipe that used to
separate the redo values from the error messages.
2) the only meaningful communications were redo requests and
I'm done.
It depends on which side is the client. Since we made the read-batch
side always the client then the errors/warnings get output to the tty
directly (for a client sender, which is the default for non-batch local
transfers, the pipe also contains any error messages from the receiver).
There is also a patch in the patches dir that sends a success message
for every correctly transferred file (delete-sent-files.diff).
I thought we could skip the redos and fake the I'm done.
Yes, it's possible to turn off the communication, which is what I was
talking about -- I was mentioning what things would need to be
special-cased.
Ah, I see what you mean about the directory tweaks.
And don't forget the hard-link post-processing -- it would also need to
happen after the receiver finished its job.
Ok, it's late and this new patch seems like the simplest solution of
all, so there's probably something very wrong with it. Basic idea:
move hard-link post-processing and directory permission tweaks from
generate_files() to recv_files(). Then there's no syncing issues,
because receiver cleans up when receiver's done. Uh, right? :)
Even with the index notification, this kind of decoupling of recv and
generator makes sense, don't you think?
Just to clarify, I don't have anything against the index notification
style gen/recv syncronization. If you think that's better, then let's
go that way.
In the future there may be a need for data to be communicated back from
the generator to the receiver (i.e. if the --fuzzy patch get improved
enough to make it worthwhile), so I think in the long run that having
the two things running in parallel (like normal) will be the easiest
thing to maintain. We can certainly revisit the issue later, but for
now I have the code in the generator sending the normal index bytes down
the output pipe (it just avoids sending the checksum data), and some
special-case read-batch code in the receiver that reads the from-the-
generator pipe and only proceeds with an update if the generator said
it was time.
One interesting thing that fell out of this was the ability to skip any
part of a batched update that had already been done. For instance, if
you run the ./BATCH.sh file and terminate it (for some reason), you can
run it again and it will skip all the already-performed updates. (One
That's a pretty compelling feature. BTW, your index notification scheme
isn't in CVS, is it?
-chris
exception: if --partial had been specified the interrupted file will
probably not update correctly, but rsync should tell you when that
happens.)
..wayne..
Index: generator.c
===
RCS file: /cvsroot/rsync/generator.c,v
retrieving revision 1.102
diff -u -r1.102 generator.c
--- generator.c 21 Jul 2004 23:59:25 - 1.102
+++ generator.c 23 Jul 2004 03:37:49 -
@@ -593,8 +593,13 @@
write_int(f_out, -1);
- if (preserve_hard_links)
- do_hard_links();
+ if (verbose 2)
+ rprintf(FINFO,generate_files finished\n);
+}
+
+void restore_directory_perms(struct file_list *flist, char *local_name)
+{
+ int i;
/* now we need to fix any directory permissions that were
* modified during the transfer */
@@ -606,6 +611,4 @@
file, i, -1);
}
- if (verbose 2)
- rprintf(FINFO,generate_files finished\n);
}
Index: receiver.c
===
RCS file: /cvsroot/rsync/receiver.c,v
retrieving revision 1.96
diff -u -r1.96 receiver.c
--- receiver.c 22 Jul 2004 15:31:06 - 1.96
+++ receiver.c 23 Jul 2004 03:37:50 -
@@ -575,12 +575,17 @@
send_msg(MSG_REDO, buf, 4);
}
}
- }
+ } /* end of while(1) */
make_backups = save_make_backups;
if (delete_after recurse !local_name flist-count 0)
delete_files(flist);
+ if (preserve_hard_links)
+ do_hard_links();
+
+ restore_directory_perms(flist, local_name);
+
if (verbose 2)
rprintf(FINFO,recv_files finished\n);
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Batch-mode rewrite
On Thu, Jul 22, 2004 at 07:01:12PM -0400, Chris Shoemaker wrote: On Thu, Jul 22, 2004 at 06:36:27PM -0700, Wayne Davison wrote: And don't forget the hard-link post-processing -- it would also need to happen after the receiver finished its job. Ok, it's late and this new patch seems like the simplest solution of all, so there's probably something very wrong with it. Basic idea: move hard-link post-processing and directory permission tweaks from generate_files() to recv_files(). Then there's no syncing issues, because receiver cleans up when receiver's done. Uh, right? :) Of course, simple isn't so great when it doesn't work. :) I see now that the hard-link post-processing looks pretty firmly attached to the generator, with the 4 hard_link_check() calls in recv_generator(). I'll look at this some more later. -chris Even with the index notification, this kind of decoupling of recv and generator makes sense, don't you think? -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync
Date: Thu Jul 22 15:30:04 2004 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv16959 Modified Files: rsync.c Log Message: The inplace handling in finish_transfer() now passes PERMS_SKIP_MTIME to set_perms() if ok_to_set_time wasn't specified. Revisions: rsync.c 1.143 = 1.144 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.c?r1=1.143r2=1.144 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Thu Jul 22 15:31:06 2004 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv17876 Modified Files: receiver.c Log Message: Should always call finish_transfer() for inplace handling, just like for keep_partial handling. Revisions: receiver.c 1.95 = 1.96 http://www.samba.org/cgi-bin/cvsweb/rsync/receiver.c?r1=1.95r2=1.96 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Thu Jul 22 15:41:09 2004 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv21137 Modified Files: rsync.yo rsync.1 Log Message: Some more --inplace improvements. Revisions: rsync.yo1.179 = 1.180 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.yo?r1=1.179r2=1.180 rsync.1 1.195 = 1.196 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.1?r1=1.195r2=1.196 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Thu Jul 22 15:43:28 2004 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv21576 Modified Files: options.c Log Message: Some --help text fixes. Revisions: options.c 1.164 = 1.165 http://www.samba.org/cgi-bin/cvsweb/rsync/options.c?r1=1.164r2=1.165 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Thu Jul 22 19:28:45 2004 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv12676 Modified Files: rsync.h io.c options.c Log Message: Changed NO_INT64 to INT64_IS_OFF_T because off_t might actually be 64 bits. The code now only complains if int64 is really too short. Revisions: rsync.h 1.207 = 1.208 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.h?r1=1.207r2=1.208 io.c1.136 = 1.137 http://www.samba.org/cgi-bin/cvsweb/rsync/io.c?r1=1.136r2=1.137 options.c 1.165 = 1.166 http://www.samba.org/cgi-bin/cvsweb/rsync/options.c?r1=1.165r2=1.166 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Thu Jul 22 22:52:39 2004 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv28026 Modified Files: backup.c Log Message: If we sucessfully renamed a file that has multiple links to it, unlink() it to ensure that rename() didn't lie to us (which it does if you try to rename() a file over another link to the same file). Revisions: backup.c1.32 = 1.33 http://www.samba.org/cgi-bin/cvsweb/rsync/backup.c?r1=1.32r2=1.33 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/testsuite
Date: Fri Jul 23 01:39:20 2004 Author: wayned Update of /data/cvs/rsync/testsuite In directory dp.samba.org:/tmp/cvs-serv32712/testsuite Modified Files: compare-dest.test Log Message: Fixed the opening comments. Revisions: compare-dest.test 1.2 = 1.3 http://www.samba.org/cgi-bin/cvsweb/rsync/testsuite/compare-dest.test?r1=1.2r2=1.3 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/testsuite
Date: Fri Jul 23 01:40:22 2004 Author: wayned Update of /data/cvs/rsync/testsuite In directory dp.samba.org:/tmp/cvs-serv418/testsuite Modified Files: README.testsuite Log Message: Chris added a missing not to a comment. Revisions: README.testsuite1.2 = 1.3 http://www.samba.org/cgi-bin/cvsweb/rsync/testsuite/README.testsuite?r1=1.2r2=1.3 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/testsuite
Date: Fri Jul 23 02:13:34 2004 Author: wayned Update of /data/cvs/rsync/testsuite In directory dp.samba.org:/tmp/cvs-serv8327/testsuite Added Files: batch-mode.test Log Message: Some basic batch-mode tests. Revisions: batch-mode.test NONE = 1.1 http://www.samba.org/cgi-bin/cvsweb/rsync/testsuite/batch-mode.test?rev=1.1 ___ rsync-cvs mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-cvs
